Ambari 使用 Knox 进行 LDAP 身份认证

跟着大数据和AI去旅行2024-01-29 23:16

Knox有如下两种认证方式：

  • ShiroProvider 对于LDAP/AD身份验证，使用用户名和密码。没有SPNEGO/Kerberos支持。
  • HadoopAuth 对于SPNEGO/Kerberos身份验证，使用委派令牌。没有LDAP/AD支持。

下面我们主要介绍下LDAP认证方式：

一 配置LDAP认证

用ambari安装的knox，默认安装目录是 /usr/hdp/current/knox-server 。默认cluster-name是default，对应的拓扑配置文件是：/usr/hdp/current/knox-server/conf/topologies/default.xml

有两种方式创建LDAP服务器，一是手工安装OpenLDAP；二是使用Knox自带的Demo LDAP：

  • 如果要手工安装OpenLDAP，参考 Centos7 下 OpenLDAP 安装。
  • 如果要使用Knox自带的DemoLDAP服务器，则在Ambari中前往 Services -> Knox -> Service Actions -> Start Demo LDAP。

下面的测试使用手工部署OpenLDAP，并在LDAP上创建了一个测试用户test（dn: cn=test,ou=users,dc=hdp,dc=com），该用户的密码是test。初学者可以安装一个JXplorer来链接LDAP服务器查看其中的数据。

为了修改Knox的默认集群拓扑文件(default.xml)，前往 Services -> Knox -> Configs -> Advanced topology，将第一个xml的第一个provider元素替换为下列内容：

xml 复制代码 
<topology>
    <gateway>
        <provider>
            <role>authentication</role>
            <name>ShiroProvider</name>
            <enabled>true</enabled>
            <param>
                <name>sessionTimeout</name>
                <value>30</value>
            </param>
            <param>
                <name>main.ldapRealm</name>
                <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
            </param>
            <param>
                <name>main.ldapRealm.userDnTemplate</name>
                <value>uid={0},ou=users,dc=hdp,dc=com</value>
            </param>
            <param>
                <name>main.ldapRealm.contextFactory.url</name>
                <value>ldap://{{knox_host_name}}</value>
            </param>
            <param>
                <name>main.ldapRealm.contextFactory.authenticationMechanism</name>
                <value>simple</value>
            </param>
            <param>
                <name>urls./**</name>
                <value>authcBasic</value>
            </param>
        </provider>
        ......
</topology>

相对于默认配置，只修改了两个参数:

xml 复制代码 
<param>
    <name>main.ldapRealm.userDnTemplate</name>
    <value>>cn={0},ou=users,dc=hdp,dc=com</value>
</param>
<param>
    <name>main.ldapRealm.contextFactory.url</name>
    <value>ldap://freeipa.testing.com</value>
</param>
  • main.ldapRealm.userDnTemplate

You would use LDAP configuration as documented above to authenticate against Active Directory as well.

Some Active Directory specific things to keep in mind:

Typical AD main.ldapRealm.userDnTemplate value looks slightly different, such as

复制代码 
cn={0},cn=users,DC=lab,DC=sample,dc=com

Please compare this with a typical Apache DS main.ldapRealm.userDnTemplate value and make note of the difference:

复制代码 
uid={0},ou=people,dc=hadoop,dc=apache,dc=org

If your AD is configured to authenticate based on just the cn and password and does not require user DN, you do not have to specify value for main.ldapRealm.userDnTemplate.

  • main.ldapRealm.contextFactory.url

你自己ldap的url，LDAP没有启用TLS，使用默认端口389。

我使用apacheds的ldap, ldap://ldap_host:10389

在ambari界面中点击Save按钮保存，并通过橙黄色按钮重启相关服务。之后会发现/usr/hdp/current/knox-server/conf/topologies/目录下的default.xml修改更新了。

二 验证Knox网关

作为一个代理网关，Knox将所有支持代理的RESTful服务和页面进行了一层地址映射

https://knox.apache.org/books/knox-1-6-0/user-guide.html#URL+Mapping

Service HA配置文档：

https://knox.apache.org/books/knox-1-6-0/user-guide.html#Default+Service+HA+support

2.1 Yarn RESTfull

现在我们来测试一下YARN的RESTful服务，打开浏览器输入https://knox_host_name:8443/gateway/default/resourcemanager/v1/cluster/apps。这是YARN查看集群运行任务的RESTful服务接口。如图下图所示，Knox网关要求进行登录认证。

输入用户名和密码进行登录认证，认证通过之后我们如愿访问到数据了，如图下图所示。

2.2 Yarn UI

接下来我们尝试访问YARN的Web UI控制台，Knox网关的默认配置中只代理了RESTful接口，所以我们需要修改它的配置文件，添加想要代理的Web UI控制台。打开Ambari找到Knox网关的配置页面，选择Advanced topology配置项，在末尾增加YARN UI的配置，保存之后需要重启Knox网关服务。

YARNUI HA配置：

xml 复制代码 
 <provider>
    <role>ha</role>
    <name>HaProvider</name>
    <enabled>true</enabled>
    <param>
        <name>WEBHDFS</name>
        <value>maxFailoverAttempts=3;failoverSleep=1000;enabled=true</value>
    </param>
    <param>
        <name>HDFSUI</name>
        <value>maxFailoverAttempts=3;failoverSleep=1000;enabled=true</value>
    </param>
    <param>
        <name>YARNUI</name>
        <value>maxFailoverAttempts=3;failoverSleep=1000;enabled=true</value>
    </param>
</provider>

<service>
    <role>YARNUI</role>
    <url>http://hdp.test.col10:8088</url>
    <url>http://hdp.test.col09:8088</url>
</service>

现在我们再次打开浏览器输入https://knox_host_name:8443/gateway/default/yarn，就能看到YARN的Web UI管理控制台了。

报错如下:

查看Knox服务日志，报错如下：

bash 复制代码 
2023-01-04 17:20:28,759 INFO  knox.gateway (AclsAuthorizationFilter.java:doFilter(104)) - Access Granted: true
2023-01-04 17:20:28,760 ERROR knox.gateway (GatewayDispatchFilter.java:isDispatchAllowed(155)) - The dispatch to http://hdp.test.col09:8088/cluster was disallowed because it fails the dispatch whitelist validation. See documentation for dispatch whitelisting.

需要修改一下gateway.dispatch.whitelist.services属性，内容里删掉YARNUI，如果不删除，则会报错：

再次访问 https://knox_host_name:8443/gateway/default/yarn ,输入用户名和密码后，访问正常

2.3 Hdfs RESTfull

配置如下：

xml 复制代码 
 <provider>
    <role>ha</role>
    <name>HaProvider</name>
    <enabled>true</enabled>
    <param>
        <name>WEBHDFS</name>
        <value>maxFailoverAttempts=3;failoverSleep=1000;enabled=true</value>
    </param>
    <param>
        <name>HDFSUI</name>
        <value>maxFailoverAttempts=3;failoverSleep=1000;enabled=true</value>
    </param>
    <param>
        <name>YARNUI</name>
        <value>maxFailoverAttempts=3;failoverSleep=1000;enabled=true</value>
    </param>
</provider>

<service>
    <role>WEBHDFS</role>
    <url>http://hdp.test.col09:50070/webhdfs</url>
    <url>http://hdp.test.col10:50070/webhdfs</url>
</service>

测试：

xml 复制代码 
curl -i -k -u gust:gust-password -X GET \
    'https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS'

2.4 HDFSUI

配置如下：

xml 复制代码 
 <provider>
    <role>ha</role>
    <name>HaProvider</name>
    <enabled>true</enabled>
    <param>
        <name>WEBHDFS</name>
        <value>maxFailoverAttempts=3;failoverSleep=1000;enabled=true</value>
    </param>
    <param>
        <name>HDFSUI</name>
        <value>maxFailoverAttempts=3;failoverSleep=1000;enabled=true</value>
    </param>
    <param>
        <name>YARNUI</name>
        <value>maxFailoverAttempts=3;failoverSleep=1000;enabled=true</value>
    </param>
</provider>

<service>
    <role>HDFSUI</role>
    <url>http://hdp.test.col09:50070</url>
    <url>http://hdp.test.col10:50070</url>
</service>

需要修改一下gateway.dispatch.whitelist.services属性，内容里删掉HDFSUI，如果不删除，则会报错：

测试：

访问https://hdp.test.col10:8443/gateway/default/hdfs 输入用户名和密码后，后台报如下错误：

bash 复制代码 
Caused by: java.io.IOException: java.io.IOException: Service connectivity error.
        at org.apache.knox.gateway.ha.dispatch.DefaultHaDispatch.failoverRequest(DefaultHaDispatch.java:125)
        at org.apache.knox.gateway.ha.dispatch.DefaultHaDispatch.executeRequest(DefaultHaDispatch.java:94)
        at org.apache.knox.gateway.ha.dispatch.DefaultHaDispatch.failoverRequest(DefaultHaDispatch.java:119)
        at org.apache.knox.gateway.ha.dispatch.DefaultHaDispatch.executeRequest(DefaultHaDispatch.java:94)
        at org.apache.knox.gateway.ha.dispatch.DefaultHaDispatch.failoverRequest(DefaultHaDispatch.java:119)
        at org.apache.knox.gateway.ha.dispatch.DefaultHaDispatch.executeRequest(DefaultHaDispatch.java:94)
        at org.apache.knox.gateway.ha.dispatch.DefaultHaDispatch.failoverRequest(DefaultHaDispatch.java:119)
        at org.apache.knox.gateway.ha.dispatch.DefaultHaDispatch.executeRequest(DefaultHaDispatch.java:94)
        at org.apache.knox.gateway.dispatch.DefaultDispatch.doGet(DefaultDispatch.java:278)
        at org.apache.knox.gateway.dispatch.GatewayDispatchFilter$GetAdapter.doMethod(GatewayDispatchFilter.java:170)
        at org.apache.knox.gateway.dispatch.GatewayDispatchFilter.doFilter(GatewayDispatchFilter.java:122)
        at org.apache.knox.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61)
        at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:372)
        at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:272)
        at org.apache.knox.gateway.filter.AclsAuthorizationFilter.doFilter(AclsAuthorizationFilter.java:108)
        at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:372)
        at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:272)
        at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.doFilterInternal(AbstractIdentityAssertionFilter.java:196)
        at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.continueChainAsPrincipal(AbstractIdentityAssertionFilter.java:153)
        at org.apache.knox.gateway.identityasserter.common.filter.CommonIdentityAssertionFilter.doFilter(CommonIdentityAssertionFilter.java:90)
        at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:372)
        at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:272)
        at org.apache.knox.gateway.filter.rewrite.api.UrlRewriteServletFilter.doFilter(UrlRewriteServletFilter.java:60)
        at org.apache.knox.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61)
        at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:372)
        at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:272)
        at org.apache.knox.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:91)
        at org.apache.knox.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:88)
        ... 82 more
Caused by: java.io.IOException: Service connectivity error.
        at org.apache.knox.gateway.dispatch.DefaultDispatch.executeOutboundRequest(DefaultDispatch.java:148)
        at org.apache.knox.gateway.ha.dispatch.DefaultHaDispatch.executeRequest(DefaultHaDispatch.java:90)
        ... 108 more
        
 
 
23/01/10 14:47:28 ||91d91611-27bc-4c0f-84f0-ece1497d461e|audit|10.252.xx.xx|HDFSUI||||access|uri|/gateway/default/hdfs|unavailable|Request method: GET
23/01/10 14:47:28 ||91d91611-27bc-4c0f-84f0-ece1497d461e|audit|10.252.xx.xx|HDFSUI|test|||authentication|uri|/gateway/default/hdfs|success|
23/01/10 14:47:28 ||91d91611-27bc-4c0f-84f0-ece1497d461e|audit|10.252.xx.xx|HDFSUI|test|||authentication|uri|/gateway/default/hdfs|success|Groups: []
23/01/10 14:47:28 ||91d91611-27bc-4c0f-84f0-ece1497d461e|audit|10.252.xx.xx|HDFSUI|test|||authorization|uri|/gateway/default/hdfs|success|
23/01/10 14:47:28 ||91d91611-27bc-4c0f-84f0-ece1497d461e|audit|10.252.xx.xx|HDFSUI|test|||dispatch|uri|https://hdp.test.col10:8443/gateway/default/hdfs?user.name=test|unavailable|Request method: GET
23/01/10 14:47:28 ||91d91611-27bc-4c0f-84f0-ece1497d461e|audit|10.252.xx.xx|HDFSUI|test|||dispatch|uri|https://hdp.test.col10:8443/gateway/default/hdfs?user.name=test|failure|
23/01/10 14:47:28 ||91d91611-27bc-4c0f-84f0-ece1497d461e|audit|10.252.xx.xx|HDFSUI|test|||access|uri|/gateway/default/hdfs|failure|

问题原因：HDP 3.1.0 - Knox proxy HDFSUI - HTTP 401 Error

this seems to be a known bug in Knox-1.0 used in HDP-3.1. One solution is to modify the URL used to access HDFS UI:

bash 复制代码 
https://<knox-host>:8443/gateway/default/hdfs?host=http://<namenode-host>:50070

Admittedly not a great solution but it worked for me. Another solution which I haven't tried is to insert the following line in the HDFSUI service definition in Knox topology file.

xml 复制代码 
<version>2.7.0</version>

This will use an older, bug-free version of HDFSUI service. More details here.

增加配置如下：

xml 复制代码 
<service>
    <role>HDFSUI</role>
    <version>2.7.0</version> 
    <url>http://hdp.test.col09:50070</url>
    <url>http://hdp.test.col10:50070</url>
</service>

再次访问https://hdp.test.col10:8443/gateway/default/hdfs，可以正常访问。

后台日志正常：

bash 复制代码 
23/01/10 15:18:23 ||e042d675-f027-4175-9dbf-12fcbe7d4a23|audit|10.252.169.246|HDFSUI||||access|uri|/gateway/default/hdfs|unavailable|Request method: GET
23/01/10 15:18:23 ||e042d675-f027-4175-9dbf-12fcbe7d4a23|audit|10.252.169.246|HDFSUI|test|||authentication|uri|/gateway/default/hdfs|success|
23/01/10 15:18:23 ||e042d675-f027-4175-9dbf-12fcbe7d4a23|audit|10.252.169.246|HDFSUI|test|||authentication|uri|/gateway/default/hdfs|success|Groups: []
23/01/10 15:18:23 ||e042d675-f027-4175-9dbf-12fcbe7d4a23|audit|10.252.169.246|HDFSUI|test|||authorization|uri|/gateway/default/hdfs|success|
23/01/10 15:18:23 ||e042d675-f027-4175-9dbf-12fcbe7d4a23|audit|10.252.169.246|HDFSUI|test|||dispatch|uri|http://hdp.test.col09:50070/?user.name=test|unavailable|Request method: GET
23/01/10 15:18:23 ||e042d675-f027-4175-9dbf-12fcbe7d4a23|audit|10.252.169.246|HDFSUI|test|||dispatch|uri|http://hdp.test.col09:50070/?user.name=test|success|Response status: 200
23/01/10 15:18:23 |||audit|10.252.169.246|HDFSUI|test|||access|uri|/gateway/default/hdfs|success|Response status: 200

2.5 HBASEUI

配置如下：

xml 复制代码 
 <provider>
    <role>ha</role>
    <name>HaProvider</name>
    <enabled>true</enabled>
    <param>
        <name>WEBHDFS</name>
        <value>maxFailoverAttempts=3;failoverSleep=1000;enabled=true</value>
    </param>
    <param>
        <name>HDFSUI</name>
        <value>maxFailoverAttempts=3;failoverSleep=1000;enabled=true</value>
    </param>
    <param>
        <name>YARNUI</name>
        <value>maxFailoverAttempts=3;failoverSleep=1000;enabled=true</value>
    </param>
    
    <param>
        <name>HBASEUI</name>
        <value>maxFailoverAttempts=3;failoverSleep=1000;enabled=true</value>
    </param>
</provider>

<service>
  <role>HBASEUI</role>
  <url>http://hdp.test.col10:16010</url>
  <url>http://hdp.test.col09:16010</url>
</service>

需要修改一下gateway.dispatch.whitelist.services属性，内容里删掉HBASEUI，如果不删除，则会报错：

访问https://hdp.test.col10:8443/gateway/default/hbase/webui 输入用户名密码即可跳到HBASE UI界面

相关推荐
TTBIGDATA21 小时前
【Atlas】Atlas Hook 消费 Kafka 报错：GroupAuthorizationException
hadoop·分布式·kafka·ambari·hdp·linq·ranger
TTBIGDATA1 天前
【Hue】Ambari开启 Kerberos 后，Hue 使用 Spark SQL出现凭证不统一问题处理
大数据·sql·spark·ambari·kerberos·hue·bigtop
TTBIGDATA2 天前
【Ranger】Ranger Admin 配置 Knox 策略时出现，the trustAnchors parameter must be non-empty
大数据·hadoop·ambari·kerberos·ranger·knox·bigtop
TTBIGDATA3 天前
【Ranger】Ambari开启Kerberos 后 ，Ranger 中 Hive 策略里，Resource lookup fail 线程池超时优化
大数据·数据仓库·hive·hadoop·ambari·hdp·ranger
TTBIGDATA3 天前
【KNOX 】服务启动后，日志中出现与 Ranger 插件资源文件相关的告警 policymgr-ssl 启动告警
大数据·网络·hdfs·ui·ambari·hdp·bigtop
TTBIGDATA4 天前
【Hue】Ambari 页面启动 Hue 失败 user ‘hadoop‘ does not exist
java·hadoop·ambari
TTBIGDATA8 天前
【Hue 】Hue 访问 YARN 返回 403 权限问题
大数据·hadoop·ambari·hdp·国产化·hue·bigtop
TTBIGDATA8 天前
【Knox】Knox 转发访问 Trino Web UI 返回 406 错误的原因与处理
大数据·运维·前端·hadoop·ui·ambari·hdp
TTBIGDATA9 天前
【knox】User: knox is not allowed to impersonate admin
大数据·运维·ambari·hdp·trino·knox·bigtop
TTBIGDATA10 天前
【Hue】Hue 访问 Hadoop 权限问题出现 403 的解决办法
大数据·hadoop·分布式·ambari·hdp·hue·bigtop
热门推荐
01GitHub 镜像站点02Claude Code + GLM4.7 避坑指南：解决 Unable to connect to Anthropic services03使用 1panel面板 部署 php网站04Vue-skills的中文文档05OpenClaw Chrome扩展使用教程 - 浏览器中继控制06让 Trae IDE 智能体 “读懂”文档 Excel+PDF+DOCX ：mcp-documents-reader 工具使用指南07从零搭建一个 PHP 登录注册系统（含完整源码）08Linux下V2Ray安装配置指南09一文了解国产算子编程语言 TileLang，TileLang 对国产开源生态的影响与启示10UV安装并设置国内源