使用 Docker Compose 搭建 Kafka 集群

本篇文章主要介绍使用 Docker Compose 搭建 Kafka 集群，实操性比较强。

Kafka在2.8.0版本之前的集群部署中必须依赖Zookeeper，因为它承担了元数据管理、Controller选举、Broker状态监控等核心协调职责；但从2.8.0版本开始，Kafka引入KRaft模式逐步替代Zookeeper，到4.0版本已完全移除该依赖。本次部署的Kafka是比较久的版本。

下面是一个 3 节点 Kafka 集群 + ZooKeeper 的 Docker Compose 配置示例：

1. 目录结构

bash 复制代码

kafka-cluster/
├── docker-compose.yml
└── .env

2. 创建 .env 配置文件

env 复制代码

# Kafka 集群配置
KAFKA_CLUSTER_NAME=my-kafka-cluster
KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181

# Broker 配置
KAFKA_BROKER_ID_1=1
KAFKA_BROKER_ID_2=2
KAFKA_BROKER_ID_3=3

# 端口配置
KAFKA_PORT_1=9092
KAFKA_PORT_2=9093
KAFKA_PORT_3=9094

# 日志目录
KAFKA_LOG_DIRS_1=/var/lib/kafka/data-1
KAFKA_LOG_DIRS_2=/var/lib/kafka/data-2
KAFKA_LOG_DIRS_3=/var/lib/kafka/data-3

3. 创建 docker-compose.yml

yaml 复制代码

version: '3.8'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.5.0
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_INIT_LIMIT: 5
      ZOOKEEPER_SYNC_LIMIT: 2
    ports:
      - "2181:2181"
    volumes:
      - zookeeper-data:/var/lib/zookeeper/data
    networks:
      - kafka-network

  kafka-1:
    image: confluentinc/cp-kafka:7.5.0
    container_name: kafka-1
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_NUM_PARTITIONS: 3
      KAFKA_DEFAULT_REPLICATION_FACTOR: 1
      KAFKA_LOG_DIRS: /var/lib/kafka/data-1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_LOG_RETENTION_HOURS: 168
    volumes:
      - kafka-data-1:/var/lib/kafka/data-1
    networks:
      - kafka-network

  kafka-2:
    image: confluentinc/cp-kafka:7.5.0
    container_name: kafka-2
    depends_on:
      - zookeeper
    ports:
      - "9093:9093"
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_NUM_PARTITIONS: 3
      KAFKA_DEFAULT_REPLICATION_FACTOR: 1
      KAFKA_LOG_DIRS: /var/lib/kafka/data-2
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_LOG_RETENTION_HOURS: 168
    volumes:
      - kafka-data-2:/var/lib/kafka/data-2
    networks:
      - kafka-network

  kafka-3:
    image: confluentinc/cp-kafka:7.5.0
    container_name: kafka-3
    depends_on:
      - zookeeper
    ports:
      - "9094:9094"
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_NUM_PARTITIONS: 3
      KAFKA_DEFAULT_REPLICATION_FACTOR: 1
      KAFKA_LOG_DIRS: /var/lib/kafka/data-3
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_LOG_RETENTION_HOURS: 168
    volumes:
      - kafka-data-3:/var/lib/kafka/data-3
    networks:
      - kafka-network

  kafka-manager:
    image: sheepkiller/kafka-manager:latest
    container_name: kafka-manager
    environment:
      ZK_HOSTS: zookeeper:2181
      APPLICATION_SECRET: "random-secret"
    ports:
      - "9000:9000"
    depends_on:
      - zookeeper
    networks:
      - kafka-network

volumes:
  zookeeper-data:
  kafka-data-1:
  kafka-data-2:
  kafka-data-3:

networks:
  kafka-network:
    driver: bridge

4. 启动集群

bash 复制代码

# 进入目录
cd kafka-cluster

# 启动所有服务
docker-compose up -d

# 查看服务状态
docker-compose ps

# 查看日志
docker-compose logs -f

5. 常用命令

测试生产者

bash 复制代码

docker exec -it kafka-1 kafka-console-producer --broker-list localhost:9092 --topic test-topic

测试消费者

bash 复制代码

docker exec -it kafka-1 kafka-console-consumer --bootstrap-server localhost:9092 --topic test-topic --from-beginning

创建 Topic

bash 复制代码

docker exec -it kafka-1 kafka-topics --create --topic my-topic \
  --bootstrap-server localhost:9092 \
  --partitions 3 \
  --replication-factor 2

查看 Topic 列表

bash 复制代码

docker exec -it kafka-1 kafka-topics --list --bootstrap-server localhost:9092

查看集群状态

bash 复制代码

docker exec -it kafka-1 kafka-topics --describe --bootstrap-server localhost:9092

6. 访问 Kafka Manager

启动后访问 http://localhost:9000 可以通过 Web UI 管理 Kafka 集群。

7. 停止集群

bash 复制代码

docker-compose down

# 删除数据卷（完全清理）
docker-compose down -v

注意：

生产环境建议增加副本数（replication-factor >= 2）
确保服务器内存充足（建议每个 Kafka 容器至少 2GB 内存）
可以通过 KAFKA_HEAP_OPTS 环境变量调整 JVM 堆内存大小