CDH-ranger
基础环境：
jdk: 1.8
maven: mvn -verison
Apache Maven 3.9.4
brew search bzip2
brew install bzip2
brew list
yum install git -y
yum install -y python3
yum install -y bzip2
yum -y install fontconfig-devel
pip3 install requests
下载ranger源码：
git clone --branch release-ranger-2.1.0 https://github.com/apache/ranger.git
########################################POM文件修改##################################
Ranger修改POM文件:
vi pom.xml
1）在repositories新增以下部分，加快编译速度。
cloudera
https://repository.cloudera.com/artifactory/cloudera-repos
true
false
2）修改组件为CDH对应的版本
<hadoop.version>3.0.0-cdh6.3.1</hadoop.version>
<hbase.version>2.1.0-cdh6.3.1</hbase.version>
<hive.version>2.1.1-cdh6.3.1</hive.version>
<kafka.version>2.2.1-cdh6.3.1</kafka.version>
<solr.version>7.4.0-cdh6.3.1</solr.version>
<zookeeper.version>3.4.5-cdh6.3.1</zookeeper.version>
主要修改包括hadoop，kafka，hbase等等。这块需要用到啥组件就改对应组件即可。
3）修改ES对应版本
<elasticsearch.version>7.13.0</elasticsearch.version>
HIVE版本兼容问题
Apache Ranger 2.1.0 对应hive版本3.1.2，CDH 6.3.2对应hive版本2.1.1，不兼容，hive server启动会报错。
1-下载Apache Ranger1.2.0 版本: git clone --branch release-ranger-1.2.0 https://github.com/apache/ranger.git
2-删除Apache Ranger 2.1.0 版本的hive插件hive-agent：rm -rf ./ranger2.1/hive-agent
3-将Apache Ranger1.2.0 版本的hive插件hive-agent拷贝到Apache Ranger 2.1.0 目录中：cp -r ./ranger1.2/hive-agent ./ranger2.1/
4-使用下面的pom文件替代hive-agent下面的pom
<?xml version="1.0" encoding="UTF-8"?> 4.0.0 ranger-hive-plugin Hive Security Plugin Hive Security Plugins jar
这里面主要做了三个操作：
1、把hive相关的版本替换成2.1.1-cdh6.3.1(按需修改)，如果不修改，默认情况下为hive-3.1.2(虽然已经在ranger的pom里面已经配置为hive.version为2.1.1-cdh6.3.1，但是没生效)
2、在添加仓库地址，之前在ranger这个父pom里面已经添加了下述的配置，但是实际没有找到
cloudera
https://repository.cloudera.com/artifactory/cloudera-repos
true
false
3、修改新拷贝过去 hive-agent目录中pom.xml的ranger版本号为2.1.0：
vim ./ranger-2.1/hive-agent/pom.xml
<parent>
<groupId>org.apache.ranger</groupId>
<artifactId>ranger</artifactId>
<version>2.1.0</version>
<relativePath>..</relativePath>
</parent>添加ranger-plugins-common依赖
在下面添加如下依赖，否则找不到添加的代码。
<dependency>
<groupId>org.apache.ranger</groupId>
<artifactId>ranger-plugins-common</artifactId>
<version>2.1.1-SNAPSHOT</version>
<scope>compile</scope>
</dependency>kylin插件POM修改
1）修改./ranger-2.1/ranger-kylin-plugin-shim/pom.xml 文件
org.apache.kylin kylin-server-base {kylin.version} provided org.apache.kylin kylin-external-htrace org.apache.calcite calcite-core org.apache.calcite calcite-linq4j 2）修改./ranger-2.1/plugin-kylin/pom.xml 文件 org.apache.kylin kylin-server-base {kylin.version} provided org.apache.kylin kylin-external-htrace org.apache.calcite calcite-core org.apache.calcite calcite-linq4j
修改distro对应的pom文件
vim ./ranger-2.1/distro/pom.xml
maven-assembly-plugin
3.3.0
主要是版本修改下，不然编译不过去。
########################################兼容性源码修改##################################
兼容性源码修改
1-修改RangerDefaultAuditHandler.java类
vim /data/packages/ranger/ranger-2.1/agents-common/src/main/java/org/apache/ranger/plugin/audit/RangerDefaultAuditHandler.java
在源码的import导入里面添加：
import org.apache.ranger.authorization.hadoop.config.RangerConfiguration;
在public class RangerDefaultAuditHandler implements行下面添加如下代码
protected static final String RangerModuleName = RangerConfiguration.getInstance().get(RangerHadoopConstants.AUDITLOG_RANGER_MODULE_ACL_NAME_PROP , RangerHadoopConstants.DEFAULT_RANGER_MODULE_ACL_NAME);
2-修改RangerConfiguration.java类：
vim /data/packages/ranger/ranger-2.1/agents-common/src/main/java/org/apache/ranger/authorization/hadoop/config/RangerConfiguration.java
在public class RangerConfiguration extends Configuration代码下面添加如下代码：
private static volatile RangerConfiguration config;
public static RangerConfiguration getInstance() {
RangerConfiguration result = config;
if (result == null) {
synchronized (RangerConfiguration.class) {
result = config;
if (result == null) {
config = result = new RangerConfiguration();
}
}
}
return result;
}
3-修改RequestUtils.java
vim /root/ranger-cdh-hylink/ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/plugin/utils/RequestUtils.java
SocketAddress socketAddress = request.getHttpChannel().getRemoteAddress();
修改为
SocketAddress socketAddress = request.getRemoteAddress();
4-修改ElasticSearchAccessAuditsService.java
vim /root/ranger-cdh-hylink/security-admin/src/main/java/org/apache/ranger/elasticsearch ElasticSearchAccessAuditsService.java
returnList.setTotalCount(response.getHits().getTotalHits().value);
修改为：
returnList.setTotalCount(response.getHits().getTotalHits());
5-修改RangerElasticsearchPlugin.java类
vim /data/packages/ranger/ranger-2.1/ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/plugin/RangerElasticsearchPlugin.java
把createComponents方法上面的@Override删除
@Override
public Collection createComponents
修改为：
public Collection createComponents
6-修改ServiceKafkaClient.java类
vim /data/packages/ranger/ranger-2.1/plugin-kafka/src/main/java/org/apache/ranger/services/kafka/client/ServiceKafkaClient.java
38行删除：
import scala.Option;
87行：
ZooKeeperClient zookeeperClient = new ZooKeeperClient(zookeeperConnect, sessionTimeout, connectionTimeout,
1, Time.SYSTEM, "kafka.server", "SessionExpireListener", Option.empty());
修改：
ZooKeeperClient zookeeperClient = new ZooKeeperClient(zookeeperConnect, sessionTimeout, connectionTimeout,
1, Time.SYSTEM, "kafka.server", "SessionExpireListener");
7-实现getHivePolicyProvider方法
vim /data/packages/ranger/ranger-2.1/hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuthorizer.java
在public boolean needTransform下面添加如下代码
@Override
public HivePolicyProvider getHivePolicyProvider() throws HiveAuthzPluginException {
if (hivePlugin == null) {
throw new HiveAuthzPluginException();
}
RangerHivePolicyProvider policyProvider = new RangerHivePolicyProvider(hivePlugin, this);
return policyProvider;
}########################################CDH平台适配##################################
CDH平台适配 - 配置文件
问题描述：
CDH在重启组件服务时为组件服务独立启动进程运行，动态生成运行配置文件目录和配置文件，ranger插件配置文件部署到CDH安装目录无法被组件服务读取到。
解决方案
在/data/packages/ranger/ranger-2.1/agents-common/src/main/java/org/apache/ranger/authorization/hadoop/config/RangerPluginConfig.java中添加copyConfigFile方法：
1）把所有的import导入的类换成如下：
import org.apache.commons.collections.CollectionUtils;
import org.apache.commons.io.FileUtils;
import org.apache.commons.io.filefilter.IOFileFilter;
import org.apache.commons.io.filefilter.RegexFileFilter;
import org.apache.commons.io.filefilter.TrueFileFilter;
import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.log4j.Logger;
import org.apache.ranger.authorization.utils.StringUtil;
import org.apache.ranger.plugin.policyengine.RangerPolicyEngineOptions;
import java.io.File;
import java.net.URL;
import java.util.*;
2）在private Set superGroups代码下面添加如下代码：
private void copyConfigFile(String serviceType) {
// 这个方法用来适配CDH版本的组件，非CDH组件需要跳出
if (serviceType.equals("presto")) {
return;
}
// 环境变量
Map map = System.getenv();
Iterator it = map.entrySet().iterator();
while (it.hasNext()) {
Map.Entry entry = (Map.Entry) it.next();
LOG.info("env key: " + entry.getKey() + ", value: " + entry.getValue());
}
// 系统变量
Properties properties = System.getProperties();
Iterator itr = properties.entrySet().iterator();
while (itr.hasNext()) {
Map.Entry entry = (Map.Entry) itr.next();
LOG.info("system key: " + entry.getKey() + ", value: " + entry.getValue());
}
String serviceHome = "CDH_" + serviceType.toUpperCase() + "_HOME";
if ("CDH_HDFS_HOME".equals(serviceHome)) {
serviceHome = "CDH_HADOOP_HOME";
}
serviceHome = System.getenv(serviceHome);
File serviceHomeDir = new File(serviceHome);
String userDir = System.getenv("CONF_DIR");
File destDir = new File(userDir);
LOG.info("-----Service Home: " + serviceHome);
LOG.info("-----User dir: " + userDir);
LOG.info("-----Dest dir: " + destDir);
IOFileFilter regexFileFilter = new RegexFileFilter("ranger-.+xml");
Collection<File> configFileList = FileUtils.listFiles(serviceHomeDir, regexFileFilter, TrueFileFilter.INSTANCE);
boolean flag = true;
for (File rangerConfigFile : configFileList) {
try {
if (serviceType.toUpperCase().equals("HIVE") && flag) {
File file = new File(rangerConfigFile.getParentFile().getPath() + "/xasecure-audit.xml");
FileUtils.copyFileToDirectory(file, destDir);
flag = false;
LOG.info("-----Source dir: " + file.getPath());
}
FileUtils.copyFileToDirectory(rangerConfigFile, destDir);
} catch (IOException e) {
LOG.error("Copy ranger config file failed.", e);
}
}}
3）在addResourcesForServiceType方法第一行添加copyConfigFile的调用：
private void addResourcesForServiceType(String serviceType) {
copyConfigFile(serviceType);
String auditCfg = "ranger-" + serviceType + "-audit.xml";
String securityCfg = "ranger-" + serviceType + "-security.xml";
String sslCfg = "ranger-policymgr-ssl.xml";
if (!addResourceIfReadable(auditCfg)) {
addAuditResource(serviceType);
}
if (!addResourceIfReadable(securityCfg)) {
addSecurityResource(serviceType);
}
if (!addResourceIfReadable(sslCfg)) {
addSslConfigResource(serviceType);
}}
CDH平台适配 - ENABLE-AGENT.SH配置
问题描述
hdfs和yarn插件安装部署后，插件jar包会部署到组件安装目录的share/hadoop/hdfs/lib子目录下，启动hdfs或yarn运行时加载不到这些jar包，会报ClassNotFoundException: Class org.apache.ranger.authorization.yarn.authorizer.RangerYarnAuthorizer not found
kafka插件安装部署后，启动运行时会从插件jar包所在目录加载ranger插件配置文件，读不到配置文件会报错addResourceIfReadable(ranger-kafka-audit.xml): couldn't find resource file location
解决方案
修改agents-common模块enable-agent.sh脚本文件：
vim /data/packages/ranger/ranger-2.1/agents-common/scripts/enable-agent.sh
将
HCOMPONENT_LIB_DIR=${HCOMPONENT_INSTALL_DIR}/share/hadoop/hdfs/lib
修改为：
HCOMPONENT_LIB_DIR=${HCOMPONENT_INSTALL_DIR}
将：
elif [ "${HCOMPONENT_NAME}" = "kafka" ]; then
HCOMPONENT_CONF_DIR=${HCOMPONENT_INSTALL_DIR}/config修改为：
elif [ "${HCOMPONENT_NAME}" = "kafka" ]; then
HCOMPONENT_CONF_DIR=${PROJ_LIB_DIR}/ranger-kafka-plugin-impl########################################源码编译##################################
源码编译
cd ./ranger2.1
编译
root@hadoop1 ranger-2.1\]# /data/maven/bin/mvn clean package install -Dmaven.test.skip=true -X
或者
\[root@hadoop1 ranger-2.1\]# /data/maven/bin/mvn clean compile package assembly:assembly install -DskipTests -Drat.skip=true
或者
\[root@hadoop1 ranger-2.1\]# /data/maven/bin/mvn clean package install -Dpmd.skip=true -Dcheckstyle.skip=true -Dmaven.test.skip=true
说明：
选择第一种方式编译，会跳过测试代码编译以及测试；
第二种方式编译会跳过测试代码测试，但是不会跳过编译；
第三种方式主要是忽略一些规范问题，比如修改源码时代码或者注释不规范，编译可能会报You have 1 PMD violation，通过这种方式解决即可。
本次使用的最后一个命令最终编译成功。构建过程躺了很多坑，差不多花了两天才解决。相比ranger编译社区版来说，要费劲一些。
报错：Could not resolve dependencies for project org.apache.ranger:ranger-kylin-plugin:jar:2.1.0
修改：ranger-release-ranger-2.1.0/ranger-kylin-plugin-shim的pom.xml文件
org.apache.kylin
kylin-server-base
k y l i n . v e r s i o n \< / v e r s i o n \> \< s c o p e \> p r o v i d e d \< / s c o p e \> \< e x c l u s i o n s \> \< e x c l u s i o n \> \< g r o u p I d \> o r g . a p a c h e . k y l i n \< / g r o u p I d \> \< a r t i f a c t I d \> k y l i n − e x t e r n a l − h t r a c e \< / a r t i f a c t I d \> \< / e x c l u s i o n \> \< e x c l u s i o n \> \< g r o u p I d \> o r g . a p a c h e . c a l c i t e \< / g r o u p I d \> \< a r t i f a c t I d \> c a l c i t e − c o r e \< / a r t i f a c t I d \> \< / e x c l u s i o n \> \< e x c l u s i o n \> \< g r o u p I d \> o r g . a p a c h e . c a l c i t e \< / g r o u p I d \> \< a r t i f a c t I d \> c a l c i t e − l i n q 4 j \< / a r t i f a c t I d \> \< / e x c l u s i o n \> \< / e x c l u s i o n s \> \< / d e p e n d e n c y \> 修改 r a n g e r − r e l e a s e − r a n g e r − 2.1.0 / p l u g i n − k y l i n 的 p o m . x m l 文件 \< d e p e n d e n c y \> \< g r o u p I d \> o r g . a p a c h e . k y l i n \< / g r o u p I d \> \< a r t i f a c t I d \> k y l i n − s e r v e r − b a s e \< / a r t i f a c t I d \> \< v e r s i o n \> {kylin.version}\ \