Windows Docker desktop 部署

Kafka

Windows Docker desktop 部署(单机版) $已验证$

注意事项：

1.使用 Hyper-V firewall 网络 IPv4 地址 172.21.192.1（插网线的情况： host.docker.internal）

yaml 复制代码

version: '3.8'

networks:
  ai_br1:
    driver: bridge

services:
  kafka:
    image: bitnami/kafka:3.5.0
    container_name: kafka
    restart: always
    ports:
      - "9092:9092"
      - "9094:9094"
    environment:
      - KAFKA_KRAFT_CLUSTER_ID=M0-lNwZsSrex6qY76fKKOg
      - KAFKA_CFG_NODE_ID=0
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@:9093
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,EXTERNAL://172.21.192.1:9094  # 使用 Hyper-V firewall 网络 IPv4 地址  172.21.192.1（插网线的情况： host.docker.internal）
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_MESSAGE_MAX_BYTES=10485760   # 设置消息最大10M
      - KAFKA_CFG_MAX_REQUEST_SIZE=10485760
      - KAFKA_NUM_PARTITIONS=2
      - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
      - KAFKA_CFG_LOG_RETENTION_HOURS=72
      - KAFKA_CFG_DELETE_RETENTION_MS=259200000
      - ALLOW_PLAINTEXT_LISTENER=yes
    volumes:
      - ./kafka_data:/bitnami/kafka
    networks:
      - ai_br1

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    restart: always
    ports:
      - "8080:8080"
    environment:
      - KAFKA_CLUSTERS_0_NAME=SingleKafkaNode
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092
      - KAFKA_CLUSTERS_0_READONLY=false
    networks:
      - ai_br1
    depends_on:
      - kafka

Kafka 3.5.0 集群搭建 $已验证$

生集群 UUID

BASH 复制代码

docker run --rm bitnami/kafka:3.5.0 kafka-storage.sh random-uuid

编写 docker-compose.yml $已验证$

yaml 复制代码

version: '3.8'

networks:
  ai_br1:
    driver: bridge

services:
  kafka0:
    image: bitnami/kafka:3.5.0
    container_name: kafka0
    restart: always
    ports:
      - "9092:9092"
      - "19094:9094"
    environment:
      - KAFKA_KRAFT_CLUSTER_ID=M0-lNwZsSrex6qY76fKKOg
      - KAFKA_CFG_NODE_ID=0
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka0:9093,1@kafka1:9093,2@kafka2:9093
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka0:9092,EXTERNAL://172.21.192.1:19094
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
      - ALLOW_PLAINTEXT_LISTENER=yes
    volumes:
      - ./kafka0_data:/bitnami/kafka
    networks:
      - ai_br1

  kafka1:
    image: bitnami/kafka:3.5.0
    container_name: kafka1
    restart: always
    ports:
      - "9093:9092"
      - "19095:9094"
    environment:
      - KAFKA_KRAFT_CLUSTER_ID=M0-lNwZsSrex6qY76fKKOg
      - KAFKA_CFG_NODE_ID=1
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka0:9093,1@kafka1:9093,2@kafka2:9093
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka1:9092,EXTERNAL://172.21.192.1:19095
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
      - ALLOW_PLAINTEXT_LISTENER=yes
    volumes:
      - ./kafka1_data:/bitnami/kafka
    networks:
      - ai_br1

  kafka2:
    image: bitnami/kafka:3.5.0
    container_name: kafka2
    restart: always
    ports:
      - "9094:9092"
      - "19096:9094"
    environment:
      - KAFKA_KRAFT_CLUSTER_ID=M0-lNwZsSrex6qY76fKKOg
      - KAFKA_CFG_NODE_ID=2
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka0:9093,1@kafka1:9093,2@kafka2:9093
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka2:9092,EXTERNAL://172.21.192.1:19096
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
      - ALLOW_PLAINTEXT_LISTENER=yes
    volumes:
      - ./kafka2_data:/bitnami/kafka
    networks:
      - ai_br1

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    restart: always
    ports:
      - "8080:8080"
    environment:
      - KAFKA_CLUSTERS_0_NAME=KafkaKRaftCluster
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka0:9092,kafka1:9092,kafka2:9092
      - KAFKA_CLUSTERS_0_READONLY=false
    networks:
      - ai_br1
    depends_on:
      - kafka0
      - kafka1
      - kafka2

升级到4.0.0 $已验证$

image: bitnami/kafka:4.0.0 主要替换版本号

其次，挂在目录不改变，只要不动 ./kafka*_data，Kafka 会自动升级 metadata 并保留所有消息。

yaml 复制代码

version: '3.8'

networks:
  ai_br1:
    driver: bridge

services:
  kafka0:
    image: bitnami/kafka:4.0.0
    container_name: kafka0
    restart: always
    ports:
      - "9092:9092"
      - "19094:9094"
    environment:
      - KAFKA_KRAFT_CLUSTER_ID=M0-lNwZsSrex6qY76fKKOg
      - KAFKA_CFG_NODE_ID=0
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka0:9093,1@kafka1:9093,2@kafka2:9093
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka0:9092,EXTERNAL://172.21.192.1:19094
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
      - ALLOW_PLAINTEXT_LISTENER=yes
    volumes:
      - ./kafka0_data:/bitnami/kafka
    networks:
      - ai_br1

  kafka1:
    image: bitnami/kafka:4.0.0
    container_name: kafka1
    restart: always
    ports:
      - "9093:9092"
      - "19095:9094"
    environment:
      - KAFKA_KRAFT_CLUSTER_ID=M0-lNwZsSrex6qY76fKKOg
      - KAFKA_CFG_NODE_ID=1
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka0:9093,1@kafka1:9093,2@kafka2:9093
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka1:9092,EXTERNAL://172.21.192.1:19095
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
      - ALLOW_PLAINTEXT_LISTENER=yes
    volumes:
      - ./kafka1_data:/bitnami/kafka
    networks:
      - ai_br1

  kafka2:
    image: bitnami/kafka:4.0.0
    container_name: kafka2
    restart: always
    ports:
      - "9094:9092"
      - "19096:9094"
    environment:
      - KAFKA_KRAFT_CLUSTER_ID=M0-lNwZsSrex6qY76fKKOg
      - KAFKA_CFG_NODE_ID=2
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka0:9093,1@kafka1:9093,2@kafka2:9093
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka2:9092,EXTERNAL://172.21.192.1:19096
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
      - ALLOW_PLAINTEXT_LISTENER=yes
    volumes:
      - ./kafka2_data:/bitnami/kafka
    networks:
      - ai_br1

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    restart: always
    ports:
      - "8080:8080"
    environment:
      - KAFKA_CLUSTERS_0_NAME=KafkaKRaftCluster
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka0:9092,kafka1:9092,kafka2:9092
      - KAFKA_CLUSTERS_0_READONLY=false
    networks:
      - ai_br1
    depends_on:
      - kafka0
      - kafka1
      - kafka2

查看升级后版本号

bash 复制代码

docker exec -it kafka0 kafka-topics.sh --version

主机 A、主机 B 和主机 C

bash 复制代码

主机 A IP: 192.168.1.10

主机 B IP: 192.168.1.11

主机 C IP: 192.168.1.12

在主机 A 上

yaml 复制代码

version: '3.8'

services:
  kafka1:
    image: bitnami/kafka:3.5.0
    container_name: kafka1
    restart: always
    ports:
      - "9094:9092" # 宿主机端口 9094 映射到容器端口 9092
    environment:
      - KAFKA_CFG_NODE_ID=1
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      # 控制器通信请使用 *实际的 IP 地址*
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@192.168.1.10:9093,2@192.168.1.11:9093,3@192.168.1.12:9093
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9092
      # 为客户端公布外部 IP。PLAINTEXT 用于此宿主机 Docker 内部的代理间通信。
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka1:9092,EXTERNAL://192.168.1.10:9094
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_MESSAGE_MAX_BYTES=10485760
      - KAFKA_CFG_MAX_REQUEST_SIZE=10485760
      - KAFKA_NUM_PARTITIONS=2
      - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
      - KAFKA_CFG_LOG_RETENTION_HOURS=72
      - KAFKA_CFG_DELETE_RETENTION_MS=259200000
      - ALLOW_PLAINTEXT_LISTENER=yes
    volumes:
      - ./kafka1_data:/bitnami/kafka
    # 每个宿主机都将有自己的桥接网络。这里的 'ai_br1' 仅限于主机 A。
    networks:
      - ai_br1

networks:
  ai_br1:
    driver: bridge

在主机 B 上

yaml 复制代码

version: '3.8'

services:
  kafka2:
    image: bitnami/kafka:3.5.0
    container_name: kafka2
    restart: always
    ports:
      - "9095:9092" # 宿主机端口 9095 映射到容器端口 9092
    environment:
      - KAFKA_CFG_NODE_ID=2
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@192.168.1.10:9093,2@192.168.1.11:9093,3@192.168.1.12:9093
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka2:9092,EXTERNAL://192.168.1.11:9095
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_MESSAGE_MAX_BYTES=10485760
      - KAFKA_CFG_MAX_REQUEST_SIZE=10485760
      - KAFKA_NUM_PARTITIONS=2
      - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
      - KAFKA_CFG_LOG_RETENTION_HOURS=72
      - KAFKA_CFG_DELETE_RETENTION_MS=259200000
      - ALLOW_PLAINTEXT_LISTENER=yes
    volumes:
      - ./kafka2_data:/bitnami/kafka
    networks:
      - ai_br1

networks:
  ai_br1:
    driver: bridge

在主机 C 上

用于 kafka3 和 kafka-ui

您可以选择任何主机运行 Kafka UI。它只需要能连接到所有 Kafka 代理即可。

yaml 复制代码

version: '3.8'

services:
  kafka3:
    image: bitnami/kafka:3.5.0
    container_name: kafka3
    restart: always
    ports:
      - "9096:9092" # 宿主机端口 9096 映射到容器端口 9092
    environment:
      - KAFKA_CFG_NODE_ID=3
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@192.168.1.10:9093,2@192.168.1.11:9093,3@192.168.1.12:9093
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka3:9092,EXTERNAL://192.168.1.12:9096
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_MESSAGE_MAX_BYTES=10485760
      - KAFKA_CFG_MAX_REQUEST_SIZE=10485760
      - KAFKA_NUM_PARTITIONS=2
      - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
      - KAFKA_CFG_LOG_RETENTION_HOURS=72
      - KAFKA_CFG_DELETE_RETENTION_MS=259200000
      - ALLOW_PLAINTEXT_LISTENER=yes
    volumes:
      - ./kafka3_data:/bitnami/kafka
    networks:
      - ai_br1

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    restart: always
    ports:
      - "8080:8080" # 映射到宿主机端口 8080
    environment:
      - KAFKA_CLUSTERS_0_NAME=MyKafkaCluster
      # Kafka UI 需要连接到所有代理的 *外部 advertised 监听器*
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=192.168.1.10:9094,192.168.1.11:9095,192.168.1.12:9096
      - KAFKA_CLUSTERS_0_READONLY=false
    # Kafka UI 在主机 C 上运行，因此可以通过其本地 ai_br1 网络连接到 kafka3 (kafka3:9092)，
    # 但需要 kafka1 和 kafka2 的外部 IP。
    networks:
      - ai_br1
    depends_on:
      - kafka3 # 仅需依赖本地 Kafka 实例以确保启动顺序，它将通过 advertised listeners 连接所有实例。

networks:
  ai_br1:
    driver: bridge

重要注意事项:
静态 IP 地址：强烈建议为您的 Kafka 主机使用静态 IP 地址。如果它们的 IP 发生变化，您的 docker-compose.yml 和 KAFKA_CFG_CONTROLLER_QUORUM_VOTERS 配置将会失效。
主机名 vs. IP 地址：虽然我为了清晰起见使用了 IP 地址，但使用可解析的主机名（例如，kafka1.yourdomain.com）通常是更好的做法。这要求您的所有主机都进行适当的 DNS 配置。如果使用主机名，请将 KAFKA_CFG_CONTROLLER_QUORUM_VOTERS 和 KAFKA_CFG_ADVERTISED_LISTENERS 中的 IP 地址替换为相应的主机名。
防火墙配置：确保您的网络防火墙允许 Kafka 所有主机之间以及任何客户端机器到 Kafka 主机之间的指定 Kafka 端口（控制器端口 9093，外部访问端口 9094、9095、9096）的流量。
数据持久化： volumes 配置 (./kafka1_data:/bitnami/kafka) 将为每个 Kafka 代理的数据创建本地目录。确保这些目录位于持久存储上并具有适当的权限。
KAFKA_CFG_ADVERTISED_LISTENERS 解析：
- PLAINTEXT://kafka1:9092：这用于同一宿主机上 Docker 网络内部的通信。如果 kafka-ui 与 kafka3 在同一宿主机上，它可以连接到 kafka3:9092。
- EXTERNAL://192.168.1.10:9094：这是公共可访问的地址，其他 Kafka 代理（用于代理间通信）和外部客户端（如 Kafka UI、生产者、消费者）将使用此地址连接到此特定的 Kafka 代理。这是多主机设置中的关键部分。
Kafka UI BOOTSTRAP_SERVERS： Kafka UI 需要配置所有 Kafka 代理的外部 advertised 监听器。这使得无论 UI 本身运行在哪里，它都能连接到整个集群。
KRaft 模式：您的配置使用了 KRaft (Kafka Raft) 模式，这意味着 Kafka 代理自行处理元数据，无需单独的 ZooKeeper 集群。这简化了部署，但使得 KAFKA_CFG_CONTROLLER_QUORUM_VOTERS 设置对于代理间通信变得更加关键。
Docker Swarm / Kubernetes：对于真正的生产级多主机部署，请考虑使用 Docker Swarm 或 Kubernetes 等编排工具。它们提供了内置的网络、服务发现和高可用性解决方案，可以简化 Kafka 等分布式应用程序的管理。

链接工具

下载地址： https://offsetexplorer.com/download.html

Windows Docker desktop 部署

Kafka

Windows Docker desktop 部署(单机版) 已验证

Kafka 3.5.0 集群搭建 已验证

升级到4.0.0 已验证

查看升级后版本号

主机 A、主机 B 和 主机 C

在主机 A 上

在主机 B 上

在主机 C 上

链接工具

Windows Docker desktop 部署(单机版) $已验证$

Kafka 3.5.0 集群搭建 $已验证$

升级到4.0.0 $已验证$

主机 A、主机 B 和主机 C