基于Docker搭建kafka集群

在单台服务器上使用 Docker Compose 部署 Kafka 3.7 KRaft 集群（3 节点） 。所有 3 个 Kafka 节点（combined mode：broker + controller）运行在同一台物理机上，通过不同端口区分，适用于开发、测试或轻量级生产场景。

特点：

单机部署，无需多台云服务器

使用官方 apache/kafka:3.7.0 镜像（内置 JDK 17）

启用 SASL/PLAIN 认证

支持从 本机外部（如本地电脑）访问 Kafka

环境要求

操作系统：CentOS 7/8/9、Ubuntu 等（本例以 CentOS 为例）
Docker ≥ 20.10
Docker Compose ≥ v2.0（或 docker-compose）
可用内存 ≥ 4GB（建议 6GB+）

目录结构

bash 复制代码

kafka-cluster/
├── docker-compose.yml
├── kafka/
│   └── kafka_server_jaas.conf
└── client/
    ├── kafka_admin_client_jaas.conf
    └── client.properties

第一步：创建配置文件

1. 创建 JAAS 文件（服务端）

bash 复制代码

mkdir -p /opt/kafka-cluster/{kafka,client}
cd /opt/kafka-cluster
cat > kafka/kafka_server_jaas.conf <<EOF
KafkaServer {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="StrongPassword123!"
    user_admin="StrongPassword123!"
    user_producer="StrongPassword456!"
    user_consumer="StrongPassword789!";
};
EOF

密码请按需修改，但所有节点必须一致。

2. 客户端认证文件（用于测试）

bash 复制代码

cat > client/kafka_admin_client_jaas.conf <<EOF
KafkaClient {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="StrongPassword123!";
};
EOF

cat > client/client.properties <<EOF
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="StrongPassword123!";
EOF

第二步：编写 `docker-compose.yml`（单机 3 节点）

复制代码

vi docker-compose.yml

内容如下：

yaml 复制代码

version: '3.8'

services:
  kafka1:
    image: apache/kafka:3.7.0
    container_name: kafka1
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: "broker,controller"
      KAFKA_LISTENERS: "SASL_PLAINTEXT://:9092,PLAINTEXT://:9094,CONTROLLER://:9093"
      KAFKA_ADVERTISED_LISTENERS: "SASL_PLAINTEXT://host.docker.internal:9092,PLAINTEXT://kafka1:9094"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "SASL_PLAINTEXT:SASL_PLAINTEXT,PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT"
      KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka1:9093,2@kafka2:9093,3@kafka3:9093"
      KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER"
      KAFKA_INTER_BROKER_LISTENER_NAME: "PLAINTEXT"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_CFG_AUTHORIZER_CLASS_NAME: "org.apache.kafka.metadata.authorizer.StandardAuthorizer"
      KAFKA_SASL_ENABLED_MECHANISMS: "PLAIN"
      KAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL: "PLAIN"
      KAFKA_OPTS: "-Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf"
    volumes:
      - ./kafka/kafka_server_jaas.conf:/opt/kafka/config/kafka_server_jaas.conf
    networks:
      - kafka-net
    extra_hosts:
      - "host.docker.internal:host-gateway"

  kafka2:
    image: apache/kafka:3.7.0
    container_name: kafka2
    ports:
      - "9093:9092"
    environment:
      KAFKA_NODE_ID: 2
      KAFKA_PROCESS_ROLES: "broker,controller"
      KAFKA_LISTENERS: "SASL_PLAINTEXT://:9092,PLAINTEXT://:9094,CONTROLLER://:9093"
      KAFKA_ADVERTISED_LISTENERS: "SASL_PLAINTEXT://host.docker.internal:9093,PLAINTEXT://kafka2:9094"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "SASL_PLAINTEXT:SASL_PLAINTEXT,PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT"
      KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka1:9093,2@kafka2:9093,3@kafka3:9093"
      KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER"
      KAFKA_INTER_BROKER_LISTENER_NAME: "PLAINTEXT"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_CFG_AUTHORIZER_CLASS_NAME: "org.apache.kafka.metadata.authorizer.StandardAuthorizer"
      KAFKA_SASL_ENABLED_MECHANISMS: "PLAIN"
      KAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL: "PLAIN"
      KAFKA_OPTS: "-Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf"
    volumes:
      - ./kafka/kafka_server_jaas.conf:/opt/kafka/config/kafka_server_jaas.conf
    networks:
      - kafka-net
    extra_hosts:
      - "host.docker.internal:host-gateway"

  kafka3:
    image: apache/kafka:3.7.0
    container_name: kafka3
    ports:
      - "9094:9092"
    environment:
      KAFKA_NODE_ID: 3
      KAFKA_PROCESS_ROLES: "broker,controller"
      KAFKA_LISTENERS: "SASL_PLAINTEXT://:9092,PLAINTEXT://:9094,CONTROLLER://:9093"
      KAFKA_ADVERTISED_LISTENERS: "SASL_PLAINTEXT://host.docker.internal:9094,PLAINTEXT://kafka3:9094"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "SASL_PLAINTEXT:SASL_PLAINTEXT,PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT"
      KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka1:9093,2@kafka2:9093,3@kafka3:9093"
      KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER"
      KAFKA_INTER_BROKER_LISTENER_NAME: "PLAINTEXT"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_CFG_AUTHORIZER_CLASS_NAME: "org.apache.kafka.metadata.authorizer.StandardAuthorizer"
      KAFKA_SASL_ENABLED_MECHANISMS: "PLAIN"
      KAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL: "PLAIN"
      KAFKA_OPTS: "-Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf"
    volumes:
      - ./kafka/kafka_server_jaas.conf:/opt/kafka/config/kafka_server_jaas.conf
    networks:
      - kafka-net
    extra_hosts:
      - "host.docker.internal:host-gateway"

networks:
  kafka-net:
    driver: bridge

关键说明：

配置	说明
`host.docker.internal`	线下想使用且没有域名的话，可以替换成公网ip(在实验时，博主是配置的公网IP)
端口映射	`9092→9092`, `9093→9092`, `9094→9092` → 外部通过 `宿主机IP:9092/9093/9094` 访问
`KAFKA_ADVERTISED_LISTENERS`	必须设为 `host.docker.internal`，否则客户端连错地址
所有节点共用同一 JAAS 文件	用户密码一致

第三步：启动集群

bash 复制代码

cd /opt/kafka-cluster
docker compose up -d

如果提示 docker compose 不存在，请安装：

bash 复制代码

sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

查看日志

bash 复制代码

docker compose logs -f kafka1

成功标志

复制代码

[main] INFO ... Kafka is now ready to serve requests

首次启动可能需要 30~60 秒完成 controller 选举。

第四步：验证集群（从宿主机外部访问）

假设你的服务器公网 IP 是 142.12.153.25，则外部可通过：

142.12.153.25:9092
142.12.153.25:9093
142.12.153.25:9094

连接 Kafka。

1. 创建 Topic（在服务器本地执行）

使用 PLAINTEXT 端口 9094 创建 topic（无需认证，内部专用）：

bash 复制代码

docker run --rm -it \
  --network kafka-cluster_kafka-net \
  apache/kafka:3.7.0 \
  /opt/kafka/bin/kafka-topics.sh \
    --create \
    --topic test-single-host \
    --partitions 3 \
    --replication-factor 3 \
    --bootstrap-server kafka1:9094,kafka2:9094,kafka3:9094
    
#列出topic
docker run --rm -it \
  --network kafka-cluster_kafka-net \
  -v $(pwd)/client/kafka_admin_client_jaas.conf:/tmp/jaas.conf \
  -v $(pwd)/client/client.properties:/tmp/client.properties \
  apache/kafka:3.7.0 bash -c "
    export KAFKA_OPTS='-Djava.security.auth.login.config=/tmp/jaas.conf' && \
    /opt/kafka/bin/kafka-topics.sh \
      --list \
      --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 \
      --command-config /tmp/client.properties
  "
  
#生产者生产消息
echo "Hello Kafka" | docker run --rm -i \
  --network kafka-cluster_kafka-net \
  -v $(pwd)/client/kafka_admin_client_jaas.conf:/tmp/jaas.conf \
  -v $(pwd)/client/client.properties:/tmp/client.properties \
  apache/kafka:3.7.0 bash -c "
    export KAFKA_OPTS='-Djava.security.auth.login.config=/tmp/jaas.conf' && \
    /opt/kafka/bin/kafka-console-producer.sh \
      --bootstrap-server kafka1:9092 \
      --producer.config /tmp/client.properties \
      --topic test-single-host
  "

#消费者消费消息
docker run --rm -it \
  --network kafka-cluster_kafka-net \
  -v $(pwd)/client/kafka_admin_client_jaas.conf:/tmp/jaas.conf \
  -v $(pwd)/client/client.properties:/tmp/client.properties \
  apache/kafka:3.7.0 bash -c "
    export KAFKA_OPTS='-Djava.security.auth.login.config=/tmp/jaas.conf' && \
    /opt/kafka/bin/kafka-console-consumer.sh \
      --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 \
      --consumer.config /tmp/client.properties \
      --topic test-single-host \
      --from-beginning
  "

2. 从本地电脑（线下）测试生产消息

bash 复制代码

# 替换 YOUR_SERVER_PUBLIC_IP
SERVER_IP="142.12.153.25"

echo "Hello from local!" | docker run --rm -i \
  -v $(pwd)/client/kafka_admin_client_jaas.conf:/tmp/jaas.conf \
  -v $(pwd)/client/client.properties:/tmp/client.properties \
  apache/kafka:3.7.0 bash -c "
    export KAFKA_OPTS='-Djava.security.auth.login.config=/tmp/jaas.conf' && \
    /opt/kafka/bin/kafka-console-producer.sh \
      --bootstrap-server 115.159.155.193:9092 \
      --producer.config /tmp/client.properties \
      --topic test-single-host
  "

3. 消费消息（本地）

bash 复制代码

docker run --rm -it \
  -v $(pwd)/client/kafka_admin_client_jaas.conf:/tmp/jaas.conf \
  -v $(pwd)/client/client.properties:/tmp/client.properties \
  apache/kafka:3.7.0 bash -c "
    export KAFKA_OPTS='-Djava.security.auth.login.config=/tmp/jaas.conf' && \
    /opt/kafka/bin/kafka-console-consumer.sh \
      --bootstrap-server ${SERVER_IP}:9092 \
      --consumer.config /tmp/client.properties \
      --topic test-single-host \
      --from-beginning \
      --timeout-ms 10000
  "

如果看到消息，说明单机 Kafka 集群工作正常！

上面23均未检验，仅在本地客户端通过过公网ip的方式连上kafka并进行生产和消费消息

安全与网络建议

防火墙：只开放

复制代码

9092-9094

给可信 IP

bash 复制代码

sudo firewall-cmd --permanent --add-port=9092-9094/tcp
sudo firewall-cmd --reload

不要用于公网生产环境：建议仅限内网或测试
升级为 SASL_SSL：如需加密传输

停止容器

bash 复制代码

docker compose down

基于Docker搭建kafka集群

环境要求

目录结构

第一步：创建配置文件

1. 创建 JAAS 文件（服务端）

2. 客户端认证文件（用于测试）

第二步：编写 docker-compose.yml（单机 3 节点）

关键说明：

第三步：启动集群

第四步：验证集群（从宿主机外部访问）

1. 创建 Topic（在服务器本地执行）

2. 从本地电脑（线下）测试生产消息

3. 消费消息（本地）

安全与网络建议

停止容器

第二步：编写 `docker-compose.yml`（单机 3 节点）