K8S yaml文件部署kafka集群（Kraft模式）——筑梦之路

筑梦之路2026-09-30 12:35

yaml文件方式，只需要对存储类名、名空间、节点IP和镜像地址进行调整，就可以在k8s部署起来的kafka集群（Kraft模式），该yaml适用于bitnami版本镜像，该yaml支持集群内外访问。

bash 复制代码 
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-headless
  namespace: test
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app: kafka
  ports:
  - name: plaintext
    port: 9092
    targetPort: 9092
  - name: controller
    port: 9093
    targetPort: 9093
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-svc
  namespace: test
spec:
  type: ClusterIP
  selector:
    statefulset.kubernetes.io/pod-name: kafka-0
  ports:
  - protocol: TCP
    port: 9092
    targetPort: 9092
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-0
  namespace: test
spec:
  type: NodePort
  selector:
    statefulset.kubernetes.io/pod-name: kafka-0
  ports:
  - protocol: TCP
    port: 39092
    targetPort: 9092
    nodePort: 39092
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-1
  namespace: test
spec:
  type: NodePort
  selector:
    statefulset.kubernetes.io/pod-name: kafka-1
  ports:
  - protocol: TCP
    port: 39093
    targetPort: 9092
    nodePort: 39093
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-2
  namespace: test
spec:
  type: NodePort
  selector:
    statefulset.kubernetes.io/pod-name: kafka-2
  ports:
  - protocol: TCP
    port: 39094
    targetPort: 9092
    nodePort: 39094
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
  namespace: test
spec:
  selector:
    matchLabels:
      app: kafka
  serviceName: kafka-headless
  replicas: 3
  updateStrategy:
    type: RollingUpdate
  podManagementPolicy: OrderedReady
  template:
    metadata:
      labels:
        app: kafka
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values: [kafka]
            topologyKey: kubernetes.io/hostname
      securityContext:
        runAsUser: 1000
        fsGroup: 1000
      containers:
      - name: kafka
        image: kafka:3.9.0-debian-12-r13
        imagePullPolicy: Always
        command:
        - bash
        - -ec
        - |
          HOSTNAME=`hostname -s`
          if [[ $HOSTNAME =~ (.*)-([0-9]+)$ ]]; then
            ORD=${BASH_REMATCH[2]}
            PORT=$((ORD + 39092))
            export KAFKA_CFG_NODE_ID="$ORD"
            export KAFKA_CFG_ADVERTISED_LISTENERS="PLAINTEXT://192.168.100.100:$PORT"
          else
            echo "Failed to get index from hostname $HOSTNAME"
            exit 1
          fi
          exec /entrypoint.sh /run.sh
        env:
        - {name: ALLOW_PLAINTEXT_LISTENER, value: 'yes'}
        - {name: KAFKA_KRAFT_CLUSTER_ID, value: 'MkU3OEVBNTcwNTJENDM2Qk'}
        - {name: KAFKA_CFG_PROCESS_ROLES, value: 'broker,controller'}
        - {name: KAFKA_CFG_LISTENERS, value: 'PLAINTEXT://:9092,CONTROLLER://:9093'}
        - {name: KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP, value: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT'}
        - {name: KAFKA_CFG_INTER_BROKER_LISTENER_NAME, value: 'PLAINTEXT'}
        - {name: KAFKA_CFG_CONTROLLER_LISTENER_NAMES, value: 'CONTROLLER'}
        - {name: KAFKA_CFG_CONTROLLER_QUORUM_VOTERS, value: '0@kafka-0.kafka-headless.test.svc.cluster.local:9093,1@kafka-1.kafka-headless.test.svc.cluster.local:9093,2@kafka-2.kafka-headless.test.svc.cluster.local:9093'}
        - {name: KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR, value: '3'}
        - {name: KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISR, value: '3'}
        - {name: KAFKA_CFG_TRANSACTION_STATE_LOG_REPLICATION_FACTOR, value: '3'}
        - {name: KAFKA_CFG_DEFAULT_REPLICATION_FACTOR, value: '3'}
        - {name: TZ, value: Asia/Shanghai}
        ports:
        - {containerPort: 9092, name: plaintext}
        - {containerPort: 9093, name: controller}
        resources:
          limits: {cpu: 500m, memory: 1Gi}
          requests: {cpu: 100m, memory: 500Mi}
        volumeMounts:
        - {name: kafka-data, mountPath: /bitnami/kafka}
  volumeClaimTemplates:
  - metadata: {name: kafka-data}
    spec:
      accessModes: [ReadWriteOnce]
      storageClassName: managed-nfs-storage
      resources:
        requests: {storage: 1000Gi}

仅供参考。

官方kafka版本镜像，apache/kakfa

bash 复制代码 
# =============================================================================
# Apache Kafka KRaft 模式 · Kubernetes StatefulSet 部署（Combined 模式 / 3 节点）
# 镜像：apache/kafka:4.3.1（Apache 官方 JVM 镜像，自 3.7.0 起提供）
#
# 拓扑：3 个 Pod 同时承担 broker + controller 角色，可容忍 1 个节点故障。
# 适用：中小规模生产、测试/预发环境、希望用最少组件跑通 KRaft 的场景。
# 用法：kubectl apply -f kafka-kraft-combined.yaml
#
# 部署前必须确认/修改：
#   1) CLUSTER_ID ------ 换成自己生成的集群 ID（见 README「部署步骤」第 1 步）
#   2) storageClassName ------ 换成集群中实际存在的 StorageClass
#   3) 存储容量、CPU/内存 requests/limits 按实际环境调整
# =============================================================================
# kafka-kraft-combined.yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: kafka
  labels:
    app.kubernetes.io/name: kafka

---
# 无头 Service：为每个 Pod 提供稳定 DNS
#   kafka-0.kafka-headless.kafka.svc.cluster.local
#   kafka-1.kafka-headless.kafka.svc.cluster.local
#   kafka-2.kafka-headless.kafka.svc.cluster.local
# publishNotReadyAddresses 必须为 true：KRaft 组建 quorum 时 Pod 之间需要互相解析，
# 若等到 Ready 才注册 DNS，会因互相解析不到而永远无法 Ready（死锁）。
apiVersion: v1
kind: Service
metadata:
  name: kafka-headless
  namespace: kafka
  labels:
    app.kubernetes.io/name: kafka
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app.kubernetes.io/name: kafka
  ports:
    - name: plaintext
      port: 9092
      targetPort: 9092
      protocol: TCP
    - name: controller
      port: 9093
      targetPort: 9093
      protocol: TCP

---
# 普通 ClusterIP Service：给集群内客户端做 bootstrap 入口
# 客户端使用 kafka-bootstrap.kafka.svc.cluster.local:9092
apiVersion: v1
kind: Service
metadata:
  name: kafka-bootstrap
  namespace: kafka
  labels:
    app.kubernetes.io/name: kafka
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: kafka
  ports:
    - name: plaintext
      port: 9092
      targetPort: 9092
      protocol: TCP

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
  namespace: kafka
  labels:
    app.kubernetes.io/name: kafka
spec:
  serviceName: kafka-headless
  replicas: 3
  # 关键：Parallel。KRaft 的 quorum 需要多个 Pod 同时起来，
  # 若用默认的 OrderedReady，kafka-0 会因 quorum 不完整而无法 Ready，
  # 导致 kafka-1/kafka-2 永远不被创建。
  podManagementPolicy: Parallel
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app.kubernetes.io/name: kafka
  template:
    metadata:
      labels:
        app.kubernetes.io/name: kafka
    spec:
      # Kafka 关停需要时间做数据落盘，留足优雅退出时间
      terminationGracePeriodSeconds: 60
      securityContext:
        runAsUser: 1000      # 官方镜像内建 appuser（uid=1000, gid=1000）
        runAsGroup: 1000
        runAsNonRoot: true
        fsGroup: 1000        # 让 PVC 挂载后对 appuser 可写，缺了会报权限错误
      # 尽量把 3 个副本打散到不同节点（软约束，节点不足时仍可调度）
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                topologyKey: kubernetes.io/hostname
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: kafka
      containers:
        - name: kafka
          image: apache/kafka:4.3.1
          imagePullPolicy: IfNotPresent
          # 官方镜像的启动命令是 /etc/kafka/docker/run（CMD，非 ENTRYPOINT，可被覆盖）。
          # 这里包一层 bash：用 Pod 名推导本节点的 node.id 与 advertised.listeners，
          # 因为这两个值每个 Pod 都不同，无法用静态 env 写死。
          # 注意：${POD_NAME##*-} 是运行时由 bash 解析的，Kubernetes 只替换 $(VAR) 形式，不会误伤。
          command:
            - bash
            - -c
            - |
              set -euo pipefail
              export KAFKA_NODE_ID="${POD_NAME##*-}"
              export KAFKA_ADVERTISED_LISTENERS="PLAINTEXT://${POD_NAME}.kafka-headless.kafka.svc.cluster.local:9092"
              exec /etc/kafka/docker/run
          ports:
            - name: plaintext
              containerPort: 9092
            - name: controller
              containerPort: 9093
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name

            # ---- 集群标识 ----------------------------------------------------
            # 示例值，务必替换为自己生成的集群 ID（kafka-storage.sh random-uuid）。
            # 同一集群所有节点必须一致；集群 ID 只在首次格式化数据目录时写入，
            # 之后想更换必须清空数据卷重新格式化。
            - name: CLUSTER_ID
              value: "4L6g3nShT-eMCtK--X86sw"

            # ---- KRaft 角色与监听 --------------------------------------------
            - name: KAFKA_PROCESS_ROLES
              value: "broker,controller"
            - name: KAFKA_LISTENERS
              value: "PLAINTEXT://:9092,CONTROLLER://:9093"
            - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
              value: "CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT"
            - name: KAFKA_INTER_BROKER_LISTENER_NAME
              value: "PLAINTEXT"
            - name: KAFKA_CONTROLLER_LISTENER_NAMES
              value: "CONTROLLER"
            # 控制器 quorum 成员：nodeId@Pod 的稳定 DNS
            - name: KAFKA_CONTROLLER_QUORUM_VOTERS
              value: "0@kafka-0.kafka-headless.kafka.svc.cluster.local:9093,1@kafka-1.kafka-headless.kafka.svc.cluster.local:9093,2@kafka-2.kafka-headless.kafka.svc.cluster.local:9093"

            # ---- 存储 --------------------------------------------------------
            - name: KAFKA_LOG_DIRS
              value: "/var/lib/kafka/data"

            # ---- 副本因子：单节点测试环境要改成 1 -----------------------------
            - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
              value: "3"
            - name: KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR
              value: "3"
            - name: KAFKA_TRANSACTION_STATE_LOG_MIN_ISR
              value: "2"
            # Kafka 4.x 新增的 share group（队列）协调器内部主题
            - name: KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR
              value: "3"
            - name: KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR
              value: "2"
            - name: KAFKA_DEFAULT_REPLICATION_FACTOR
              value: "3"
            - name: KAFKA_MIN_INSYNC_REPLICAS
              value: "2"
            - name: KAFKA_NUM_PARTITIONS
              value: "3"

            # ---- 常规参数 ----------------------------------------------------
            - name: KAFKA_AUTO_CREATE_TOPICS_ENABLE
              value: "false"
            - name: KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS
              value: "0"
            - name: KAFKA_LOG_RETENTION_HOURS
              value: "168"
            # JVM 堆：按容器内存 limits 的 1/2 左右设置，留出页缓存和直接内存
            - name: KAFKA_HEAP_OPTS
              value: "-Xms1g -Xmx1g"

          # 官方镜像里 bin 脚本齐全，用 broker API 探测更贴近"真的能服务请求"，
          # 比单纯 TCP 端口探测更可靠。
          startupProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - /opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 >/dev/null 2>&1
            initialDelaySeconds: 20
            periodSeconds: 10
            timeoutSeconds: 10
            failureThreshold: 30
          readinessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - /opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 >/dev/null 2>&1
            periodSeconds: 10
            timeoutSeconds: 10
            failureThreshold: 6
          # 刻意不配 livenessProbe：Kafka 启动慢且重启不能自愈（多为 quorum/存储问题），
          # 探活失败重启只会造成滚动失败循环，靠 startupProbe + readinessProbe 已足够。
          resources:
            requests:
              cpu: "1"
              memory: "2Gi"
            limits:
              cpu: "2"
              memory: "4Gi"
          volumeMounts:
            - name: data
              mountPath: /var/lib/kafka/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: standard   # TODO 改成集群实际的 StorageClass
        resources:
          requests:
            storage: 20Gi

---
# 滚动升级/节点维护时保证最多只有 1 个 Kafka Pod 不可用
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: kafka
  namespace: kafka
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: kafka
bash 复制代码 
# =============================================================================
# 可选：把 Kafka 暴露到集群外（NodePort 方案）
# 镜像：apache/kafka:4.3.1（Apache 官方 JVM 镜像）
#
# 原理：客户端连接 broker 后会拿到 broker 自报的 advertised.listeners 地址并直连，
#      所以每个 broker 必须有"自己的"对外入口，不能共用一个 Service/NodePort，
#      否则客户端会被转发到错误的 broker。因此这里给每个 Pod 建一个独立 Service，
#      用 selector: statefulset.kubernetes.io/pod-name 精确绑定到单个 Pod，
#      并固定 nodePort，使地址可预测。
#
# 使用前需要同时修改 StatefulSet（kafka-kraft-combined.yaml）：
#
# 1) 增加一个 EXTERNAL 监听端口与安全协议映射：
#      - name: KAFKA_LISTENERS
#        value: "PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094"
#      - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
#        value: "CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT"
#
# 2) 增加容器端口 9094（containerPort: 9094, name: external）
#
# 3) 增加下行 API 环境变量，拿到 Pod 所在节点 IP：
#      - name: NODE_IP
#        valueFrom:
#          fieldRef:
#            fieldPath: status.hostIP
#
# 4) 改写 command 中的 advertised.listeners，把 EXTERNAL 指向"节点 IP + 固定 nodePort"
#    （nodePort 与下方 Service 的 30092/30093/30094 一一对应）：
#      export KAFKA_ADVERTISED_LISTENERS="PLAINTEXT://${POD_NAME}.kafka-headless.kafka.svc.cluster.local:9092,EXTERNAL://${NODE_IP}:$((30092 + ${POD_NAME##*-}))"
#
# 5) 客户端 bootstrap 用 <任一节点IP>:30092,<另一节点IP>:30093,<第三节点IP>:30094
#    若集群前面有负载均衡/LB，建议把三个 nodePort 都注册到同一个对外地址上。
#
# 安全提醒：PLAINTEXT 暴露到公网等于无认证无加密，生产环境务必改用
#          SASL_SSL / SSL 监听（镜像支持挂载 /etc/kafka/secrets 下的证书与 JAAS 配置）。
# =============================================================================

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-external-0
  namespace: kafka
  labels:
    app.kubernetes.io/name: kafka
    app.kubernetes.io/component: external
spec:
  type: NodePort
  selector:
    statefulset.kubernetes.io/pod-name: kafka-0
  ports:
    - name: external
      port: 9094
      targetPort: 9094
      nodePort: 30092
      protocol: TCP

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-external-1
  namespace: kafka
  labels:
    app.kubernetes.io/name: kafka
    app.kubernetes.io/component: external
spec:
  type: NodePort
  selector:
    statefulset.kubernetes.io/pod-name: kafka-1
  ports:
    - name: external
      port: 9094
      targetPort: 9094
      nodePort: 30093
      protocol: TCP

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-external-2
  namespace: kafka
  labels:
    app.kubernetes.io/name: kafka
    app.kubernetes.io/component: external
spec:
  type: NodePort
  selector:
    statefulset.kubernetes.io/pod-name: kafka-2
  ports:
    - name: external
      port: 9094
      targetPort: 9094
      nodePort: 30094
      protocol: TCP
bash 复制代码 
# =============================================================================
# Apache Kafka KRaft 模式 · Kubernetes StatefulSet 部署（Isolated 模式）
# 镜像：apache/kafka:4.3.1（Apache 官方 JVM 镜像）
#
# 拓扑：3 个专用 controller + 3 个专用 broker（角色分离），共 6 个 Pod。
# 适用：生产环境。controller 只跑元数据共识，不受 broker 流量影响，职责清晰、可独立扩缩容。
# 用法：kubectl apply -f kafka-kraft-isolated.yaml
#       （与 kafka-kraft-combined.yaml 二选一，不要同时部署）
#
# 部署前必须确认/修改：
#   1) CLUSTER_ID、2) storageClassName、3) 容量与资源配置
# =============================================================================
# 

---
apiVersion: v1
kind: Namespace
metadata:
  name: kafka
  labels:
    app.kubernetes.io/name: kafka

# ============================== Controller 层 ==============================

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-controller-headless
  namespace: kafka
  labels:
    app.kubernetes.io/name: kafka
    app.kubernetes.io/component: controller
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app.kubernetes.io/name: kafka
    app.kubernetes.io/component: controller
  ports:
    - name: controller
      port: 9093
      targetPort: 9093
      protocol: TCP

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka-controller
  namespace: kafka
  labels:
    app.kubernetes.io/name: kafka
    app.kubernetes.io/component: controller
spec:
  serviceName: kafka-controller-headless
  replicas: 3
  podManagementPolicy: Parallel   # quorum 组建要求多个 Pod 同时启动
  selector:
    matchLabels:
      app.kubernetes.io/name: kafka
      app.kubernetes.io/component: controller
  template:
    metadata:
      labels:
        app.kubernetes.io/name: kafka
        app.kubernetes.io/component: controller
    spec:
      terminationGracePeriodSeconds: 60
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        runAsNonRoot: true
        fsGroup: 1000
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                topologyKey: kubernetes.io/hostname
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: kafka
                    app.kubernetes.io/component: controller
      containers:
        - name: kafka
          image: apache/kafka:4.3.1
          imagePullPolicy: IfNotPresent
          # controller 节点只需要 node.id；注意 controller-only 节点严禁设置
          # KAFKA_ADVERTISED_LISTENERS（镜像启动脚本会直接报错退出）。
          command:
            - bash
            - -c
            - |
              set -euo pipefail
              export KAFKA_NODE_ID="${POD_NAME##*-}"
              exec /etc/kafka/docker/run
          ports:
            - name: controller
              containerPort: 9093
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: CLUSTER_ID
              value: "4L6g3nShT-eMCtK--X86sw"   # TODO 替换为自己生成的集群 ID
            - name: KAFKA_PROCESS_ROLES
              value: "controller"
            - name: KAFKA_LISTENERS
              value: "CONTROLLER://:9093"
            - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
              value: "CONTROLLER:PLAINTEXT"
            - name: KAFKA_CONTROLLER_LISTENER_NAMES
              value: "CONTROLLER"
            - name: KAFKA_CONTROLLER_QUORUM_VOTERS
              value: "0@kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9093,1@kafka-controller-1.kafka-controller-headless.kafka.svc.cluster.local:9093,2@kafka-controller-2.kafka-controller-headless.kafka.svc.cluster.local:9093"
            - name: KAFKA_LOG_DIRS
              value: "/var/lib/kafka/data"
            # controller 也承载元数据日志，保留策略可略短于业务数据
            - name: KAFKA_LOG_RETENTION_HOURS
              value: "168"
            - name: KAFKA_HEAP_OPTS
              value: "-Xms1g -Xmx1g"
          # controller 节点不提供 broker 端口，用 TCP 探测其 CONTROLLER 监听端口；
          # 更精确的 quorum 状态可用 kafka-metadata-quorum.sh 人工检查（见部署手册「验证」）。
          startupProbe:
            tcpSocket:
              port: 9093
            initialDelaySeconds: 20
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 30
          readinessProbe:
            tcpSocket:
              port: 9093
            periodSeconds: 15
            timeoutSeconds: 5
            failureThreshold: 6
          resources:
            requests:
              cpu: "500m"
              memory: "2Gi"
            limits:
              cpu: "1"
              memory: "4Gi"
          volumeMounts:
            - name: data
              mountPath: /var/lib/kafka/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: standard   # TODO 改成集群实际的 StorageClass
        resources:
          requests:
            storage: 10Gi

# =============================== Broker 层 ================================

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-broker-headless
  namespace: kafka
  labels:
    app.kubernetes.io/name: kafka
    app.kubernetes.io/component: broker
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app.kubernetes.io/name: kafka
    app.kubernetes.io/component: broker
  ports:
    - name: plaintext
      port: 9092
      targetPort: 9092
      protocol: TCP

---
# 客户端 bootstrap 入口：kafka-bootstrap.kafka.svc.cluster.local:9092
apiVersion: v1
kind: Service
metadata:
  name: kafka-bootstrap
  namespace: kafka
  labels:
    app.kubernetes.io/name: kafka
    app.kubernetes.io/component: broker
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: kafka
    app.kubernetes.io/component: broker
  ports:
    - name: plaintext
      port: 9092
      targetPort: 9092
      protocol: TCP

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka-broker
  namespace: kafka
  labels:
    app.kubernetes.io/name: kafka
    app.kubernetes.io/component: broker
spec:
  serviceName: kafka-broker-headless
  replicas: 3
  podManagementPolicy: Parallel
  selector:
    matchLabels:
      app.kubernetes.io/name: kafka
      app.kubernetes.io/component: broker
  template:
    metadata:
      labels:
        app.kubernetes.io/name: kafka
        app.kubernetes.io/component: broker
    spec:
      terminationGracePeriodSeconds: 60
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        runAsNonRoot: true
        fsGroup: 1000
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                topologyKey: kubernetes.io/hostname
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: kafka
                    app.kubernetes.io/component: broker
      containers:
        - name: kafka
          image: apache/kafka:4.3.1
          imagePullPolicy: IfNotPresent
          # broker 的 node.id 从 3 开始，避免与 controller 的 0/1/2 冲突
          command:
            - bash
            - -c
            - |
              set -euo pipefail
              export KAFKA_NODE_ID="$(( ${POD_NAME##*-} + 3 ))"
              export KAFKA_ADVERTISED_LISTENERS="PLAINTEXT://${POD_NAME}.kafka-broker-headless.kafka.svc.cluster.local:9092"
              exec /etc/kafka/docker/run
          ports:
            - name: plaintext
              containerPort: 9092
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: CLUSTER_ID
              value: "4L6g3nShT-eMCtK--X86sw"   # TODO 与 controller 保持一致
            - name: KAFKA_PROCESS_ROLES
              value: "broker"
            - name: KAFKA_LISTENERS
              value: "PLAINTEXT://:9092"
            - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
              value: "CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT"
            - name: KAFKA_INTER_BROKER_LISTENER_NAME
              value: "PLAINTEXT"
            - name: KAFKA_CONTROLLER_LISTENER_NAMES
              value: "CONTROLLER"
            - name: KAFKA_CONTROLLER_QUORUM_VOTERS
              value: "0@kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9093,1@kafka-controller-1.kafka-controller-headless.kafka.svc.cluster.local:9093,2@kafka-controller-2.kafka-controller-headless.kafka.svc.cluster.local:9093"
            - name: KAFKA_LOG_DIRS
              value: "/var/lib/kafka/data"
            - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
              value: "3"
            - name: KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR
              value: "3"
            - name: KAFKA_TRANSACTION_STATE_LOG_MIN_ISR
              value: "2"
            - name: KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR
              value: "3"
            - name: KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR
              value: "2"
            - name: KAFKA_DEFAULT_REPLICATION_FACTOR
              value: "3"
            - name: KAFKA_MIN_INSYNC_REPLICAS
              value: "2"
            - name: KAFKA_NUM_PARTITIONS
              value: "3"
            - name: KAFKA_AUTO_CREATE_TOPICS_ENABLE
              value: "false"
            - name: KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS
              value: "0"
            - name: KAFKA_LOG_RETENTION_HOURS
              value: "168"
            - name: KAFKA_HEAP_OPTS
              value: "-Xms2g -Xmx2g"
          startupProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - /opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 >/dev/null 2>&1
            initialDelaySeconds: 20
            periodSeconds: 10
            timeoutSeconds: 10
            failureThreshold: 30
          readinessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - /opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 >/dev/null 2>&1
            periodSeconds: 10
            timeoutSeconds: 10
            failureThreshold: 6
          resources:
            requests:
              cpu: "1"
              memory: "4Gi"
            limits:
              cpu: "2"
              memory: "8Gi"
          volumeMounts:
            - name: data
              mountPath: /var/lib/kafka/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: standard   # TODO 改成集群实际的 StorageClass
        resources:
          requests:
            storage: 50Gi

---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: kafka-controller
  namespace: kafka
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: kafka
      app.kubernetes.io/component: controller

---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: kafka-broker
  namespace: kafka
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: kafka
      app.kubernetes.io/component: broker
相关推荐
极小狐2 小时前
CI 作业里 kubectl 连不上集群？用 Kubernetes Agent 打通部署链路的 7 个步骤
ci/cd·kubernetes·gitlab·devops·k8s部署
站在墙头上2 小时前
ubantu安装docker
docker·容器
坏脾气的小十七3 小时前
docker基本知识
docker·容器·eureka
guo_wen_qiang4 小时前
jenkins流水线参数化配置
运维·docker·容器·jenkins·持续部署
打工仔折腾 AI4 小时前
Prometheus接入Pushgateway实战：二进制与Docker部署、指标推送与远程写入
后端·python·docker·容器·性能优化·prometheus·ai agent 实战
xianyuCcCcCCCcc4 小时前
Docker 容器与网络
网络·docker·容器
筑梦之路4 小时前
Kafka KRaft 模式 Kubernetes 部署手册（StatefulSet + apache/kafka 官方镜像）——筑梦之路
kafka·kubernetes·apache
A.说学逗唱的Coke5 小时前
【云原生专题】Kubernetes 备份完全实战：用 Velero 搞定集群备份、恢复与跨集群迁移（附完整命令与踩坑记录）
云原生·容器·kubernetes
誰能久伴不乏16 小时前
保姆级 Docker 入门教程
docker·容器·eureka
热门推荐
01国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）02历年考研数学一、数学二、数学三真题试卷及答案PDF03Telegram APK 下载安装完整指南 — 2026年最新04GitHub 镜像站点05蜘蛛磁力 搜索引擎大全，如何使用蜘蛛磁力查找磁力链接062026 爱知・名古屋亚运会怎么看？直播平台+赛程安排分享07微信历史版本含下载地址（ Windows PC | 安卓 | MAC ）及设置微信不更新08jev-ultrafast 深度解析：7 秒订机票的浏览器 Agent 是如何炼成的09给AI一张“业务概念地图“：本体如何从哲学走向企业智能10Windows 10/11 如何退出微软账户并改用本地账户登录（含找不到选项的解决方法）