Apache Knox 2.0.0使用

向他一样rap2024-05-12 19:37

目录

介绍

使用

gateway-site.xml

users.ldif

my_hdfs.xml

my_yarn.xml

其它

介绍

The Apache Knox Gateway is a system that provides a single point of authentication and access for Apache Hadoop services in a cluster. The goal is to simplify Hadoop security for both users (i.e. who access the cluster data and execute jobs) and operators (i.e. who control access and manage the cluster). The gateway runs as a server (or cluster of servers) that provide centralized access to one or more Hadoop clusters. In general the goals of the gateway are as follows:

  • Provide perimeter security for Hadoop REST APIs to make Hadoop security easier to setup and use
    • Provide authentication and token verification at the perimeter
    • Enable authentication integration with enterprise and cloud identity management systems
    • Provide service level authorization at the perimeter
  • Expose a single URL hierarchy that aggregates REST APIs of a Hadoop cluster
    • Limit the network endpoints (and therefore firewall holes) required to access a Hadoop cluster
    • Hide the internal Hadoop cluster topology from potential attackers

使用

解压后，目录如下：

进入conf目录:

文件说明：

gateway-site.xml

网关文件，修改如下:

XML 复制代码 
    <property>
        <name>gateway.port</name>
        <value>18483</value>
        <description>The HTTP port for the Gateway.</description>
    </property>
    <property>
        <name>gateway.path</name>
        <value>my/mimi</value>
        <description>The default context path for the gateway.</description>
    </property>
    <property>
        <name>gateway.dispatch.whitelist</name>
        <value>.*$</value>
    </property>

users.ldif

用户文件，全部如下

XML 复制代码 
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

version: 1

# Please replace with site specific values
dn: dc=hadoop,dc=apache,dc=org
objectclass: organization
objectclass: dcObject
o: Hadoop
dc: hadoop

# Entry for a sample people container
# Please replace with site specific values
dn: ou=people,dc=hadoop,dc=apache,dc=org
objectclass:top
objectclass:organizationalUnit
ou: people

dn: ou=myhadoop,dc=hadoop,dc=apache,dc=org
objectclass:top
objectclass:organizationalUnit
ou: myhadoop

# Entry for a sample end user
# Please replace with site specific values
dn: uid=guest,ou=people,dc=hadoop,dc=apache,dc=org
objectclass:top
objectclass:person
objectclass:organizationalPerson
objectclass:inetOrgPerson
cn: Guest
sn: User
uid: guest
userPassword:123456

dn: uid=myclient,ou=people,dc=hadoop,dc=apache,dc=org
objectclass:top
objectclass:person
objectclass:organizationalPerson
objectclass:inetOrgPerson
cn: Myclient
sn: Client
uid: myclient
userPassword:qwe

dn: uid=myhdfs,ou=myhadoop,dc=hadoop,dc=apache,dc=org
objectclass:top
objectclass:person
objectclass:organizationalPerson
objectclass:inetOrgPerson
cn: myhdfs
sn: myhdfs
uid: myhdfs
userPassword:123456

dn: uid=myyarn,ou=myhadoop,dc=hadoop,dc=apache,dc=org
objectclass:top
objectclass:person
objectclass:organizationalPerson
objectclass:inetOrgPerson
cn: myyarn
sn: myyarn
uid: myyarn
userPassword:123


# entry for sample user admin
dn: uid=admin,ou=people,dc=hadoop,dc=apache,dc=org
objectclass:top
objectclass:person
objectclass:organizationalPerson
objectclass:inetOrgPerson
cn: Admin
sn: Admin
uid: admin
userPassword:123456

dn: uid=root,ou=myhadoop,dc=hadoop,dc=apache,dc=org
objectclass:top
objectclass:person
objectclass:organizationalPerson
objectclass:inetOrgPerson
cn: Admin
sn: Admin
uid: root
userPassword:123456

# entry for sample user sam
dn: uid=sam,ou=people,dc=hadoop,dc=apache,dc=org
objectclass:top
objectclass:person
objectclass:organizationalPerson
objectclass:inetOrgPerson
cn: sam
sn: sam
uid: sam
userPassword:sam-password

# entry for sample user tom
dn: uid=tom,ou=people,dc=hadoop,dc=apache,dc=org
objectclass:top
objectclass:person
objectclass:organizationalPerson
objectclass:inetOrgPerson
cn: tom
sn: tom
uid: tom
userPassword:tom-password

# create FIRST Level groups branch
dn: ou=groups,dc=hadoop,dc=apache,dc=org
objectclass:top
objectclass:organizationalUnit
ou: groups
description: generic groups branch

# create the analyst group under groups
dn: cn=analyst,ou=groups,dc=hadoop,dc=apache,dc=org
objectclass:top
objectclass: groupofnames
cn: analyst
description:analyst  group
member: uid=sam,ou=people,dc=hadoop,dc=apache,dc=org
member: uid=tom,ou=people,dc=hadoop,dc=apache,dc=org
member: uid=myhdfs,ou=myhadoop,dc=hadoop,dc=apache,dc=org
member: uid=myyarn,ou=myhadoop,dc=hadoop,dc=apache,dc=org

# create the scientist group under groups
dn: cn=scientist,ou=groups,dc=hadoop,dc=apache,dc=org
objectclass:top
objectclass: groupofnames
cn: scientist
description: scientist group
member: uid=sam,ou=people,dc=hadoop,dc=apache,dc=org

# create the admin group under groups
dn: cn=admin,ou=groups,dc=hadoop,dc=apache,dc=org
objectclass:top
objectclass: groupofnames
cn: admin
description: admin group
member: uid=admin,ou=people,dc=hadoop,dc=apache,dc=org
member: uid=root,ou=myhadoop,dc=hadoop,dc=apache,dc=org

注意，我在这里面添加了两个用户 myhdfs，myyarn

进入knox-2.0.0/conf/topologies 文件，会发现一个sandbox.xml文件

它是一个模板文件，使用时改一个名字，我的如下

my_hdfs.xml

XML 复制代码 
<?xml version="1.0" encoding="UTF-8"?>
<topology>
   <name>my_hdfs</name>
   <gateway>
      <provider>
         <role>authentication</role>
         <name>ShiroProvider</name>
         <enabled>true</enabled>
         <param>
            <name>sessionTimeout</name>
            <value>30</value>
         </param>
         <param>
            <name>main.ldapRealm</name>
            <value>org.apache.knox.gateway.shirorealm.KnoxLdapRealm</value>
         </param>
         <param>
            <name>main.ldapContextFactory</name>
            <value>org.apache.knox.gateway.shirorealm.KnoxLdapContextFactory</value>
         </param>
         <param>
            <name>main.ldapRealm.contextFactory</name>
            <value>$ldapContextFactory</value>
         </param>


		<param>
			<name>main.ldapRealm.contextFactory.systemUsername</name>
			<value>uid=myclient,ou=people,dc=hadoop,dc=apache,dc=org</value>
		</param>
		<param>
			<name>main.ldapRealm.contextFactory.systemPassword</name>
			<value>${ALIAS=ldcSystemPassword}</value>
		</param>
		<param>
			<name>main.ldapRealm.userSearchBase</name>
			<value>ou=myhadoop,dc=hadoop,dc=apache,dc=org</value>
		</param>
		<param>
			<name>main.ldapRealm.userSearchAttributeName</name>
			<value>uid</value>
		</param>
		<param>
			<name>main.ldapRealm.userSearchFilter</name>
			<value>(&amp;(objectclass=person)(uid={0})(uid=myhdfs))</value>
		</param>

         <param>
            <name>main.ldapRealm.contextFactory.url</name>
            <value>ldap://localhost:33389</value>
         </param>
         <param>
            <name>main.ldapRealm.contextFactory.authenticationMechanism</name>
            <value>simple</value>
         </param>
         <param>
            <name>urls./**</name>
            <value>authcBasic</value>
         </param>
      </provider>
      <provider>
         <role>identity-assertion</role>
         <name>Default</name>
         <enabled>true</enabled>
      </provider>
      <provider>
         <role>hostmap</role>
         <name>static</name>
         <enabled>true</enabled>
         <param>
            <name>localhost</name>
            <value>my_hdfs</value>
         </param>
      </provider>
   </gateway>
<!--
   <service>
      <role>NAMENODE</role>
      <url>hdfs://hadoop02:9000</url>
   </service>
   <service>
      <role>WEBHDFS</role>
      <url>http://hadoop02:50070/webhdfs</url>
   </service>
-->   
   <service>
      <role>HDFSUI</role>
      <url>http://hadoop02:50070</url>
      <version>2.7.0</version>
   </service>

</topology>

my_yarn.xml

XML 复制代码 
<?xml version="1.0" encoding="UTF-8"?>
<topology>
   <name>my_yarn</name>
   <gateway>
      <provider>
         <role>authentication</role>
         <name>ShiroProvider</name>
         <enabled>true</enabled>
         <param>
            <name>sessionTimeout</name>
            <value>30</value>
         </param>
         <param>
            <name>main.ldapRealm</name>
            <value>org.apache.knox.gateway.shirorealm.KnoxLdapRealm</value>
         </param>
         <param>
            <name>main.ldapContextFactory</name>
            <value>org.apache.knox.gateway.shirorealm.KnoxLdapContextFactory</value>
         </param>
         <param>
            <name>main.ldapRealm.contextFactory</name>
            <value>$ldapContextFactory</value>
         </param>
         
		 <!-- 登录测试 -->
        <param>
			<name>main.ldapRealm.contextFactory.systemUsername</name>
			<value>uid=myclient,ou=people,dc=hadoop,dc=apache,dc=org</value>
		</param>
		<param>
			<name>main.ldapRealm.contextFactory.systemPassword</name>
			<value>${ALIAS=ldcSystemPassword}</value>
		</param>
		<param>
			<name>main.ldapRealm.userSearchBase</name>
			<value>ou=myhadoop,dc=hadoop,dc=apache,dc=org</value>
		</param>
		<param>
			<name>main.ldapRealm.userSearchAttributeName</name>
			<value>uid</value>
		</param>
		<param>
			<name>main.ldapRealm.userSearchFilter</name>
			<value>(&amp;(objectclass=person)(uid={0})(uid=myyarn))</value>
		</param>
		 <!-- 安全测试-->
		<param><name>csrf.enabled</name><value>true</value></param>
		<param><name>csrf.customHeader</name><value>X-XSRF-Header</value></param>
		<param><name>csrf.methodsToIgnore</name><value>GET,OPTIONS,HEAD</value></param>
		<param><name>cors.enabled</name><value>false</value></param>
		<param><name>xframe.options.enabled</name><value>true</value></param>
		<param><name>xss.protection.enabled</name><value>true</value></param>
		<param><name>strict.transport.enabled</name><value>true</value></param>
		<param><name>rate.limiting.enabled</name><value>true</value></param>

         <param>
            <name>main.ldapRealm.contextFactory.url</name>
            <value>ldap://localhost:33389</value>
         </param>
         <param>
            <name>main.ldapRealm.contextFactory.authenticationMechanism</name>
            <value>simple</value>
         </param>
         <param>
            <name>urls./**</name>
            <value>authcBasic</value>
         </param>
      </provider>
      <provider>
         <role>identity-assertion</role>
         <name>Default</name>
         <enabled>true</enabled>
      </provider>
      <provider>
         <role>hostmap</role>
         <name>static</name>
         <enabled>true</enabled>
         <param>
            <name>localhost</name>
            <value>my_yarn</value>
         </param>
      </provider>
   </gateway>
   <service>
      <role>YARNUI</role>
      <url>http://hadoop03:8088</url>
   </service>
   <service>
    <role>JOBHISTORYUI</role>
    <url>http://hadoop03:19888</url>
  </service>

</topology>

说明，配置中的${ALIAS=ldcSystemPassword}是生成的密文

参考：

如果测试不成功改名实际密码测试

之后进入knox-2.0.0/bin

执行(注意，不能使用root用户)

XML 复制代码 
./ldap.sh start
./knoxcli.sh create-master
./gateway.sh start

网站访问：

https://192.168.200.11:18483/my/mimi/my_yarn/yarn

账号：myyarn 密码 123

https://192.168.200.11:18483/my/mimi/my_hdfs/hdfs

账号：myhdfs 密码 123456

其它

通过我的多次测试，成功实现了，不同用户访问不同页面。下面是几个问题

1 必须有多个单独的xml才能实现不同用户访问不同页面

2 我测试成功的只有hdfs、yarn、hbase。之后准备测试hue，发现死活不成功

注意：文中xml单用户访问页面可以直接复制，然后修改。官网描述很多测试失败。

相关推荐
HashData酷克数据3 小时前
Apache Cloudberry 向量化实践（二）：如何识别和定位向量化系统的性能瓶颈？
apache
HashData酷克数据1 天前
流批一体的“奥卡姆剃刀”：Apache Cloudberry 增量物化视图应用解析
apache
lang201509281 天前
什么是Apache Ignite的affinity（亲和性）
apache
Apache IoTDB2 天前
Apache IoTDB V2.0.4 发布｜新增用户自定义表函数及多种内置表函数功能
apache·iotdb
TracyCoder1232 天前
Apache Shiro 框架详解
安全·apache·shiro·认证·登录
SelectDB2 天前
Apache Doris 实时更新技术揭秘：为何在 OLAP 领域表现卓越？
数据库·数据分析·apache
__只是为了好玩__3 天前
Apache http 强制 https
http·https·apache·ssl
云游4 天前
利用外部Postgresql及zookeeper，启动Apache Dolphinscheduler3.1.9
分布式·postgresql·zookeeper·apache·工作流任务调度
Aurora_NeAr5 天前
Apache Iceberg数据湖基础
apache
源图客5 天前
Apache Nutch介绍与部署编译
apache·nutch
热门推荐
01全球最强模型Grok4，国内已可免费使用！（附教程）02KGG转MP3工具|非KGM文件|解密音频03Coze扣子平台完整体验和实践（附国内和国际版对比）04浅谈几种js设计模式05集群聊天服务器---MySQL数据库的建立06扣子（coze）实战|我用扣子搭建了一个自动分析小红薯笔记内容的AI应用|详细步骤拆解07【无标题】08使用Ruby接入实时行情API教程09DeepSeek各版本说明与优缺点分析10Java学习第十五部分——MyBatis