【kafka】docker容器bitnami/kafka使用SASL鉴权(无TLS)

背景

最近在学习kafka消息队列,了解到kafka是通过SASL来进行用户认证的。起初,因为btinami/kafka官方的一段内容让我以为SASL和TLS是绑定使用的,导致心思花在解决TLS配置上去,官方原文如下:

You must also use your own certificates for SSL. You can drop your Java Key Stores or PEM files into /opt/bitnami/kafka/config/certs. If the JKS or PEM certs are password protected (recommended), you will need to provide it to get access to the keystores:

后来发现其实是可以单独使用SASL,而且官方文档没有完整的配置资料,最后在Issue上找到解决方案。

过程

1.编写一个包含SASL配置的docker-compose.yml文件

yml 复制代码
version: "2"
services:
  kafka:
    image: 'bitnami/kafka:latest'
    hostname: localhost
    ports:
      - '9092:9092'
    environment:
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_NODE_ID=0
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CLIENT_LISTENER_NAME=CLIENT
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=CLIENT://:9092,CONTROLLER://:9093
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,CLIENT:SASL_PLAINTEXT
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka:9093
      - KAFKA_CFG_SASL_ENABLED_MECHANISMS=SCRAM-SHA-512
      - KAFKA_CLIENT_USERS=test
      - KAFKA_CLIENT_PASSWORDS=123456

将容器跑起来看日志,发现报错:

Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: inter.broker.listener.name must be a listener name defined in advertised.listeners. The valid options based on currently configured listeners are CLIENT

需要配置inter.broker.listener.name 参数,去官网看一下介绍。

以下是inter.broker.listener.name 参数的官方描述:

Name of listener used for communication between brokers. If this is unset, the listener name is defined by security.inter.broker.protocol. It is an error to set this and security.inter.broker.protocol properties at the same time.

以下是security.inter.broker.protocol参数的官方描述:

Security protocol used to communicate between brokers. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL. It is an error to set this and inter.broker.listener.name properties at the same time.

以下是对两个参数关系的一些官方描述:

Inter-broker listener must be configured using the static broker configuration inter.broker.listener.name or security.inter.broker.protocol.
Among the listeners in this list, it is possible to declare the listener to be used for inter-broker communication by setting the inter.broker.listener.name configuration to the name of the listener. The primary purpose of the inter-broker listener is partition replication. If not defined, then the inter-broker listener is determined by the security protocol defined by security.inter.broker.protocol, which defaults to PLAINTEXT.

由上述可知,inter.broker.listener.namesecurity.inter.broker.protocol 参数的作用是一致的,都是为了设置broker之间通信的配置,由于security.inter.broker.protocol设置之后依然存在问题,本文暂不讨论,后续就以inter.broker.listener.name 参数进行处理。

2.设置inter.broker.listener.name

inter.broker.listener.name 可以设置为一个新的listener,也可以设置为现有的listener(controller除外),本文就以设置现有的listener为例,将改值设置为CLIENT:

yml 复制代码
version: "2"
services:
  kafka:
    image: 'bitnami/kafka:latest'
    hostname: localhost
    ports:
      - '9092:9092'
    environment:
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=CLIENT
      # ... 和上面的配置一样

此时把容器跑起来,会得到以下错误:

ERROR ==> When using SASL for inter broker comunication the mechanism should be provided using KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL

只需要设置KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL,可选值为PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 。在官网介绍中,其实应该还有GSSAPI默认值的,但是这个值在容器使用中会出现错误,而且在bitnami/kafka文档中也没有写支持改值,因此不作讨论。

3.设置KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL

KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL设置为PLAINSCRAM-SHA-256SCRAM-SHA-512

yml 复制代码
version: "2"
services:
  kafka:
    image: 'bitnami/kafka:latest'
    hostname: localhost
    ports:
      - '9092:9092'
    environment:
      - KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL=SCRAM-SHA-512
      # ... 和上面的配置一样

这里需要注意是,KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL 的值需要在KAFKA_CFG_SASL_ENABLED_MECHANISMS 范围内,否则你将看到以下错误:

Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: sasl.mechanism.inter.broker.protocol must be included in sasl.enabled.mechanisms when SASL is used for inter-broker communication

KAFKA_CFG_SASL_ENABLED_MECHANISMS 支持多个值,以 , 隔开即可,如:
KAFKA_CFG_SASL_ENABLED_MECHANISMS=PLAIN,SCRAM-SHA-512

至此,一个简单的SASL的docker-compose.yml配置就完成了

完整配置文件

yml 复制代码
version: "2"
services:
  kafka:
    image: 'bitnami/kafka:latest'
    hostname: localhost
    ports:
      - '9092:9092'
    environment:
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_NODE_ID=0
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CLIENT_LISTENER_NAME=CLIENT
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=CLIENT
      - KAFKA_CFG_LISTENERS=CLIENT://:9092,CONTROLLER://:9093
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,CLIENT:SASL_PLAINTEXT
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka:9093
      - KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL=SCRAM-SHA-512
      - KAFKA_CFG_SASL_ENABLED_MECHANISMS=SCRAM-SHA-512
      - KAFKA_CLIENT_USERS=test
      - KAFKA_CLIENT_PASSWORDS=123456

参考文章

相关推荐
曾经的三心草1 小时前
小程序项目-购物-首页与准备
运维·服务器·小程序·项目·购物
小园子的小菜2 小时前
RocketMQ中的NameServer主要数据结构
java·中间件·rocketmq·java-rocketmq
对你无可奈何5 小时前
从Proxmox VE开始:安装与配置指南
运维·服务器·devops
lingllllove6 小时前
使用 HTTP::Server::Simple 实现轻量级 HTTP 服务器
服务器·网络协议·http
Linux运维老纪6 小时前
K8s之Service详解(Detailed Explanation of K8s Service)
服务器·网络·云原生·容器·kubernetes·云计算·运维开发
神秘剑客_CN7 小时前
使用windows笔记本让服务器上网
运维·服务器·windows
黑牛先生9 小时前
【Linux】动静态库
linux·运维·服务器
明 庭13 小时前
通过 Docker 部署 pSQL 服务器的教程
服务器·docker·容器
身在江湖的郭大侠15 小时前
Linux内核
linux·服务器
破-风17 小时前
linux的用法
linux·运维·服务器