目录
1、修改官方包中的build_ranger_using_docker.sh
一、Ranger简介
Apache Ranger提供一个集中式安全管理框架, 并解决授权和审计。它可以对Hadoop生态的组件如HDFS、Yarn、Hive、Hbase等进行细粒度的数据访问控制。通过操作Ranger控制台,管理员可以轻松的通过配置策略来控制用户访问权限。
1、组件列表
| # | Service Name | Listen Port | Core Ranger Service |
|---|---|---|---|
| 1 | ranger | 6080/tcp | Y (ranger engine - 3.0.0-SNAPSHOT version) |
| 2 | ranger-postgres | 5432/tcp | Y (ranger datastore) |
| 3 | ranger-solr | 8983/tcp | Y (audit store) |
| 4 | ranger-zk | 2181/tcp | Y (used by solr) |
| 5 | ranger-usersync | - | Y (user/group synchronization from Local Linux/Mac) |
| 6 | ranger-kms | 9292/tcp | N (needed only for Encrypted Storage / TDE) |
| 7 | ranger-tagsync | - | N (needed only for Tag Based Policies to be sync from ATLAS) |
2、支持的数据引擎服务
| # | Service Name | Listen Port | Service Description |
|---|---|---|---|
| 1 | Hadoop | 8088/tcp 9000/tcp | Apache Hadoop 3.3.0 Protected by Apache Ranger's Hadoop Plugin |
| 2 | HBase | 16000/tcp 16010/tcp 16020/tcp 16030/tcp | Apache HBase 2.4.6 Protected by Apache Ranger's HBase Plugin |
| 3 | Hive | 10000/tcp | Apache Hive 3.1.2 Protected by Apache Ranger's Hive Plugin |
| 4 | Kafka | 6667/tcp | Apache Kafka 2.8.1 Protected by Apache Ranger's Kafka Plugin |
| 5 | Knox | 8443/tcp | Apache Knox 1.4.0 Protected by Apache Ranger's Knox Plugin |
二、主机环境准备
1、关闭防火墙
systemctl stop firewalld.service
systemctl disable firewalld.service
2、关闭SELINUX
sed -i.bak$DATE '/^SELINUX=/c SELINUX=disabled' /etc/selinux/config
setenforce 0
3、安装docker
yum install -y docker
systemctl start docker
systemctl enable docker
4、下载Ranger源码包
Apache Ranger官网没有可以直接部署的安装包，必须通过源码进行编译。
官网地址：Apache Ranger - Download Apache Ranger?
wget https://www.apache.org/dist/ranger/2.4.0/apache-ranger-2.4.0.tar.gz --no-check-certificate
5、下载Maven安装包
wget https://dlcdn.apache.org/maven/maven-3/3.9.4/binaries/apache-maven-3.9.4-bin.tar.gz --no-check-certificate
三、编译Ranger源码
1、修改官方包中的build_ranger_using_docker.sh
#!/bin/bash
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License
#This script creates the Docker image (if not already created) and runs maven in the container
#1. Install Docker
#2. Checkout Ranger source and go to the root directory
#3. Run this script. If host is linux, then run this script as "sudo $0 ..."
#4. If you are running on Mac, then you don't need to use "sudo"
#5. To delete the image, run "[sudo] docker rmi ranger_dev"
#Usage: [sudo] ./build_ranger_using_docker.sh [-build_image] mvn <build params>
#Example 1 (default no param): (mvn -Pall -DskipTests=true clean compile package install)
#Example 2 (Regular build): ./build_ranger_using_docker.sh mvn -Pall clean install -DskipTests=true
#Example 3 (Recreate Docker image): ./build_ranger_using_docker.sh mvn -Pall -build_image clean install -DskipTests=true
#Notes: To remove build image manually, run "docker rmi ranger_dev" or "sudo docker rmi ranger_dev"
default_command="mvn -Pall -DskipTests=true clean compile package install"
build_image=0
if [ "$1" = "-build_image" ]; then
build_image=1
shift
fi
params=$*
if [ $# -eq 0 ]; then
params=$default_command
fi
image_name="ranger_dev"
remote_home="$HOME"
container_name="--name ranger_build"
if [ ! -d security-admin ]; then
echo "ERROR: Run the script from root folder of source. e.g. $HOME/git/ranger"
exit 1
fi
images=`docker images | cut -f 1 -d " "`
[[ $images =~ $image_name ]] && found_image=1 || build_image=1
if [ $build_image -eq 1 ]; then
echo "Creating image $image_name ..."
docker rmi -f $image_name
docker build -t $image_name - <<Dockerfile
FROM centos:centos7.6.1810
RUN mkdir /tools
WORKDIR /tools
#Install default services
RUN yum install -y wget git gcc bzip2 fontconfig python3 java-1.8.0-openjdk-devel.x86_64
RUN ln -sf /usr/bin/python3 /usr/bin/python
ENV JAVA_HOME /usr/lib/jvm/java-1.8.0-openjdk/
ENV PATH JAVA_HOME/bin:PATH
RUN wget https://dlcdn.apache.org/maven/maven-3/3.9.4/binaries/apache-maven-3.9.4-bin.tar.gz --no-check-certificate
RUN tar xfz apache-maven-3.9.4-bin.tar.gz
RUN ln -sf /tools/apache-maven-3.9.4 /tools/maven
ENV PATH /tools/maven/bin:$PATH
ENV MAVEN_OPTS "-Xmx2048m -XX:MaxPermSize=512m"
RUN mkdir -p /scripts
RUN echo "#!/bin/bash" > /scripts/mvn.sh
RUN echo 'set -x; exec "\$@" ' >> /scripts/mvn.sh
RUN chmod -R 777 /scripts
RUN chmod -R 777 /tools
ENTRYPOINT ["/scripts/mvn.sh"]
Dockerfile
fi
src_folder=`pwd`
LOCAL_M2="$HOME/.m2"
mkdir -p $LOCAL_M2
set -x
docker run --rm -v "{src_folder}:/ranger" -w "/ranger" -v "{LOCAL_M2}:${remote_home}/.m2" $container_name $image_name $params
说明：
考虑实验运行环境为centos7.6，且国内有部分外国源访问不到，所以做了一定修改和裁剪，本次实验中，使用root用户运行该脚本，容器内使用root用户进行编译操作。
1、原脚本的基础镜像为centos:lastest，其对应Centos 8.1，现修改为centos:centos7.6.1810；
2、原脚本安装jkd8时，使用AWS s3's docker-assets里的jdk-8u101-linux-x64.rpm，现修改为使用centos自带的openjdk1.8，即java-1.8.0-openjdk-devel.x86_64；
3、原脚本未安装python3，最终编译时会找不到python3包而报错，现增加安装python3，同时设置默认使用python3，即RUN ln -sf /usr/bin/python3 /usr/bin/python
4、原脚本安装maven时，使用ADD来获取apache-maven-3.6.3-bin.tar.gz并校验包，现修改为使用wget获得最新的apache-maven-3.9.4-bin.tar.gz，且不做额外的包正确性校验，即wget https://dlcdn.apache.org/maven/maven-3/3.9.4/binaries/apache-maven-3.9.4-bin.tar.gz --no-check-certificatewget
5、原脚本的启动脚本创建并使用了非root用户builder，但会与后面运行容器时映射本地卷组有权限限制，考虑到只是临时编译使用，剔除所有builder用户的内容，包含gosu安装、用户创建、用户判断等，只保留echo 'set -x; exec "\$@" ' >> /scripts/mvn.sh
6、原脚本中{remote_home}为空值，会将运行该脚本的用户Home下的.m2映射到容器内根目录的.m2，现修改为容器内工作用户的Home目录，即remote_home="HOME"
2、运行脚本编译
chmod +x build_ranger_using_docker.sh
./build_ranger_using_docker.sh
说明：
参照脚本使用说明
#Usage: [sudo] ./build_ranger_using_docker.sh [-build_image] mvn <build params>
#Example 1 (default no param): (mvn -Pall -DskipTests=true clean compile package install)
#Example 2 (Regular build): ./build_ranger_using_docker.sh mvn -Pall clean install -DskipTests=true
#Example 3 (Recreate Docker image): ./build_ranger_using_docker.sh mvn -Pall -build_image clean install -DskipTests=true
3、编译检查
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for ranger 2.4.0:
[INFO]
[INFO] ranger ............................................. SUCCESS [ 12.567 s]
[INFO] Jdbc SQL Connector ................................. SUCCESS [ 13.553 s]
[INFO] Credential Support ................................. SUCCESS [ 14.914 s]
[INFO] Audit Component .................................... SUCCESS [01:09 min]
[INFO] ranger-plugin-classloader .......................... SUCCESS [ 9.662 s]
[INFO] Common library for Plugins ......................... SUCCESS [02:03 min]
[INFO] ranger-intg ........................................ SUCCESS [ 40.185 s]
[INFO] Installer Support Component ........................ SUCCESS [ 8.196 s]
[INFO] Credential Builder ................................. SUCCESS [ 12.157 s]
[INFO] Embedded Web Server Invoker ........................ SUCCESS [ 33.355 s]
[INFO] Key Management Service ............................. SUCCESS [01:40 min]
[INFO] HBase Security Plugin Shim ......................... SUCCESS [ 52.109 s]
[INFO] HBase Security Plugin .............................. SUCCESS [01:25 min]
[INFO] Hdfs Security Plugin ............................... SUCCESS [ 36.159 s]
[INFO] Hive Security Plugin ............................... SUCCESS [ 41.491 s]
[INFO] Knox Security Plugin Shim .......................... SUCCESS [ 9.255 s]
[INFO] Knox Security Plugin ............................... SUCCESS [ 21.750 s]
[INFO] Storm Security Plugin .............................. SUCCESS [ 16.017 s]
[INFO] YARN Security Plugin ............................... SUCCESS [ 13.554 s]
[INFO] Ozone Security Plugin .............................. SUCCESS [ 12.752 s]
[INFO] Ranger Util ........................................ SUCCESS [ 11.776 s]
[INFO] Unix Authentication Client ......................... SUCCESS [ 11.990 s]
[INFO] User Group Synchronizer Util ....................... SUCCESS [ 6.909 s]
[INFO] Security Admin Web Application ..................... SUCCESS [08:54 min]
[INFO] KAFKA Security Plugin .............................. SUCCESS [01:17 min]
[INFO] SOLR Security Plugin ............................... SUCCESS [01:18 min]
[INFO] NestedStructure Security Plugin .................... SUCCESS [ 24.474 s]
[INFO] NiFi Security Plugin ............................... SUCCESS [ 12.265 s]
[INFO] NiFi Registry Security Plugin ...................... SUCCESS [ 11.211 s]
[INFO] Presto Security Plugin ............................. SUCCESS [ 24.201 s]
[INFO] Kudu Security Plugin ............................... SUCCESS [ 14.920 s]
[INFO] Unix User Group Synchronizer ....................... SUCCESS [02:08 min]
[INFO] Ldap Config Check Tool ............................. SUCCESS [ 11.640 s]
[INFO] Unix Authentication Service ........................ SUCCESS [ 11.348 s]
[INFO] KMS Security Plugin ................................ SUCCESS [01:13 min]
[INFO] Tag Synchronizer ................................... SUCCESS [ 45.784 s]
[INFO] Hdfs Security Plugin Shim .......................... SUCCESS [ 9.535 s]
[INFO] Hive Security Plugin Shim .......................... SUCCESS [01:23 min]
[INFO] YARN Security Plugin Shim .......................... SUCCESS [ 42.092 s]
[INFO] OZONE Security Plugin Shim ......................... SUCCESS [ 23.710 s]
[INFO] Storm Security Plugin shim ......................... SUCCESS [ 10.665 s]
[INFO] KAFKA Security Plugin Shim ......................... SUCCESS [ 10.838 s]
[INFO] SOLR Security Plugin Shim .......................... SUCCESS [ 22.091 s]
[INFO] Atlas Security Plugin Shim ......................... SUCCESS [ 28.752 s]
[INFO] KMS Security Plugin Shim ........................... SUCCESS [ 52.920 s]
[INFO] Presto Security Plugin Shim ........................ SUCCESS [ 26.065 s]
[INFO] ranger-examples .................................... SUCCESS [ 0.272 s]
[INFO] Ranger Examples - Conditions and ContextEnrichers .. SUCCESS [ 11.692 s]
[INFO] Ranger Examples - SampleApp ........................ SUCCESS [ 5.863 s]
[INFO] Ranger Examples - Ranger Plugin for SampleApp ...... SUCCESS [ 10.167 s]
[INFO] sample-client ...................................... SUCCESS [ 11.777 s]
[INFO] Apache Ranger Examples Distribution ................ SUCCESS [ 6.742 s]
[INFO] Ranger Tools ....................................... SUCCESS [ 35.518 s]
[INFO] Atlas Security Plugin .............................. SUCCESS [ 41.615 s]
[INFO] SchemaRegistry Security Plugin ..................... SUCCESS [03:02 min]
[INFO] Sqoop Security Plugin .............................. SUCCESS [ 53.693 s]
[INFO] Sqoop Security Plugin Shim ......................... SUCCESS [ 14.680 s]
[INFO] Kylin Security Plugin .............................. SUCCESS [03:33 min]
[INFO] Kylin Security Plugin Shim ......................... SUCCESS [ 41.171 s]
[INFO] Elasticsearch Security Plugin Shim ................. SUCCESS [ 22.381 s]
[INFO] Elasticsearch Security Plugin ...................... SUCCESS [ 37.204 s]
[INFO] Apache Ranger Distribution ......................... SUCCESS [02:26 min]
[INFO] Unix Native Authenticator .......................... SUCCESS [ 4.438 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 49:17 min
[INFO] Finished at: 2023-08-07T10:43:31Z
[INFO] ------------------------------------------------------------------------
在target目录可以看到生成的程序包：
-rw-r--r-- 1 root root 579387182 Aug 7 18:42 ranger-2.4.0-admin.tar.gz
-rw-r--r-- 1 root root 43729654 Aug 7 18:43 ranger-2.4.0-atlas-plugin.tar.gz
-rw-r--r-- 1 root root 34172214 Aug 7 18:43 ranger-2.4.0-elasticsearch-plugin.tar.gz
-rw-r--r-- 1 root root 39122941 Aug 7 18:42 ranger-2.4.0-hbase-plugin.tar.gz
-rw-r--r-- 1 root root 37684529 Aug 7 18:42 ranger-2.4.0-hdfs-plugin.tar.gz
-rw-r--r-- 1 root root 37478412 Aug 7 18:42 ranger-2.4.0-hive-plugin.tar.gz
-rw-r--r-- 1 root root 56846325 Aug 7 18:42 ranger-2.4.0-kafka-plugin.tar.gz
-rw-r--r-- 1 root root 195376717 Aug 7 18:43 ranger-2.4.0-kms.tar.gz
-rw-r--r-- 1 root root 51454934 Aug 7 18:42 ranger-2.4.0-knox-plugin.tar.gz
-rw-r--r-- 1 root root 36625366 Aug 7 18:43 ranger-2.4.0-kylin-plugin.tar.gz
-rw-r--r-- 1 root root 34201 Aug 7 18:43 ranger-2.4.0-migration-util.tar.gz
-rw-r--r-- 1 root root 43393403 Aug 7 18:42 ranger-2.4.0-ozone-plugin.tar.gz
-rw-r--r-- 1 root root 57425250 Aug 7 18:43 ranger-2.4.0-presto-plugin.tar.gz
-rw-r--r-- 1 root root 16563346 Aug 7 18:43 ranger-2.4.0-ranger-tools.tar.gz
-rw-r--r-- 1 root root 36915 Aug 7 18:42 ranger-2.4.0-solr_audit_conf.tar.gz
-rw-r--r-- 1 root root 38256335 Aug 7 18:42 ranger-2.4.0-solr-plugin.tar.gz
-rw-r--r-- 1 root root 36860763 Aug 7 18:43 ranger-2.4.0-sqoop-plugin.tar.gz
-rw-r--r-- 1 root root 6376456 Aug 7 18:43 ranger-2.4.0-src.tar.gz
-rw-r--r-- 1 root root 51760282 Aug 7 18:42 ranger-2.4.0-storm-plugin.tar.gz
-rw-r--r-- 1 root root 31046503 Aug 7 18:42 ranger-2.4.0-tagsync.tar.gz
-rw-r--r-- 1 root root 20128101 Aug 7 18:42 ranger-2.4.0-usersync.tar.gz
-rw-r--r-- 1 root root 35792990 Aug 7 18:42 ranger-2.4.0-yarn-plugin.tar.gz
参考文档：
Ranger Installation Guide - Ranger - Apache Software Foundation