Docker + CentOS 部署 Zookeeper 集群 + Kubernetes Operator 自动化运维方案

环境说明

主机 IP	主机名	节点角色	数据目录	Kubernetes 节点标签
192.168.10.100	zk1	Master	`/opt/zookeeper/data`	zk-cluster=true
192.168.10.101	zk2	Worker	`/opt/zookeeper/data`	zk-cluster=true
192.168.10.102	zk3	Worker	`/opt/zookeeper/data`	zk-cluster=true
192.168.10.103	zk4	Worker	`/opt/zookeeper/data`	zk-cluster=true
192.168.10.104	zk5	Worker	`/opt/zookeeper/data`	zk-cluster=true

一、基础环境部署（所有节点）

1. 系统配置

bash

复制代码

# 设置主机名
sudo hostnamectl set-hostname zk1  # 分别在每台机器执行对应主机名

# 编辑hosts文件
sudo tee -a /etc/hosts <<EOF 192.168.10.100 zk1 192.168.10.101 zk2 192.168.10.102 zk3 192.168.10.103 zk4 192.168.10.104 zk5 EOF # 关闭SELinux sudo setenforce 0 sudo sed -i 's/^SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config # 优化内核参数 sudo tee -a /etc/sysctl.conf <<EOF net.core.somaxconn=65535 net.ipv4.tcp_max_syn_backlog=65535 vm.swappiness=1 EOF sudo sysctl -p

2. Docker 安装

bash

复制代码

# 安装依赖
sudo dnf install -y yum-utils device-mapper-persistent-data lvm2

# 添加Docker仓库
sudo yum-config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo # 安装Docker sudo dnf install -y docker-ce docker-ce-cli containerd.io # 配置Docker sudo mkdir -p /etc/docker sudo tee /etc/docker/daemon.json <<EOF { "log-driver": "json-file", "log-opts": { "max-size": "100m", "max-file": "3" } } EOF # 启动Docker sudo systemctl start docker sudo systemctl enable docker

3. Kubernetes 组件安装

bash

复制代码

# 禁用Swap
sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab # 安装kubeadm/kubelet/kubectl cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF sudo dnf install -y kubelet kubeadm kubectl --disableexcludes=kubernetes sudo systemctl enable --now kubelet # 初始化Master节点 (仅在zk1执行) sudo kubeadm init --pod-network-cidr=10.244.0.0/16 \ --control-plane-endpoint="zk1:6443" \ --upload-certs \ --apiserver-advertise-address=192.168.10.100 # 配置kubectl (在zk1执行) mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config # 安装网络插件 kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml # 加入Worker节点 (在zk2-zk5执行) # 使用kubeadm init输出的join命令 kubeadm join zk1:6443 --token <token> --discovery-token-ca-cert-hash <hash>

二、Zookeeper Operator 部署

1. 安装 Zookeeper Operator

bash

复制代码

# 创建命名空间
kubectl create ns zookeeper-operator

# 部署Operator
kubectl apply -f https://raw.githubusercontent.com/pravega/zookeeper-operator/master/deploy/all_ns/rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/pravega/zookeeper-operator/master/deploy/all_ns/operator.yaml

# 验证Operator状态
kubectl get pods -n zookeeper-operator

2. 创建 Zookeeper 集群 CRD

zookeeper-cluster.yaml:
yaml

复制代码

apiVersion: zookeeper.pravega.io/v1beta1
kind: ZookeeperCluster
metadata: name: zookeeper-cluster namespace: default spec: replicas: 5 image: repository: zookeeper tag: 3.8.0 persistence: storageClassName: local-storage volumeReclaimPolicy: Retain size: 20Gi config: initLimit: 15 syncLimit: 5 tickTime: 2000 autopurge: snapRetainCount: 10 purgeInterval: 24 pod: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - zookeeper topologyKey: kubernetes.io/hostname nodeSelector: zk-cluster: "true" securityContext: runAsUser: 1000 fsGroup: 1000 resources: requests: memory: "2Gi" cpu: "1" limits: memory: "4Gi" cpu: "2" security: enable: true jaasConfig: secretRef: zk-jaas-secret tlsConfig: enable: true secretRef: zk-tls-secret metrics: enable: true port: 7000

3. 创建安全配置

bash

复制代码

# JAAS 认证配置
kubectl create secret generic zk-jaas-secret \
  --from-literal=jaas-config="Server {
    org.apache.zookeeper.server.auth.DigestLoginModule required
    user_admin=\"adminpassword\" user_appuser=\"apppassword\"; };" # TLS 证书配置 # (提前生成keystore.jks) kubectl create secret generic zk-tls-secret \ --from-file=keystore.jks=keystore.jks \ --from-literal=keystore-password=changeit

4. 创建存储类

local-storage.yaml:
yaml

复制代码

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: name: local-storage provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer

5. 部署集群

bash

复制代码

kubectl apply -f local-storage.yaml
kubectl apply -f zookeeper-cluster.yaml

# 查看集群状态
kubectl get zookeepercluster
kubectl get pods -l app=zookeeper

三、自动化运维功能实现

1. 自动扩缩容

bash

复制代码

# 水平扩展
kubectl patch zk zookeeper-cluster --type='merge' -p '{"spec":{"replicas":7}}' # 垂直扩容 kubectl patch zk zookeeper-cluster --type='merge' -p '{"spec":{"pod":{"resources":{"limits":{"memory":"8Gi"}}}}}'

2. 自动备份与恢复

zk-backup-job.yaml:
yaml

复制代码

apiVersion: batch/v1
kind: CronJob
metadata: name: zk-backup spec: schedule: "0 2 * * *" jobTemplate: spec: template: spec: containers: - name: backup image: zookeeper:3.8.0 command: ["/bin/sh", "-c"] args: - | echo "Connecting to ${ZK_SERVER}" echo "savemn" | nc ${ZK_SERVER} 2181 tar czf /backup/$(date +%Y%m%d).tar.gz -C /data . volumeMounts: - name: backup-volume mountPath: /backup - name: data-volume mountPath: /data restartPolicy: OnFailure volumes: - name: backup-volume persistentVolumeClaim: claimName: zk-backup-pvc - name: data-volume persistentVolumeClaim: claimName: $(ZK_PVC)

3. 自动监控告警

prometheus-monitoring.yaml:
yaml

复制代码

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata: name: zookeeper-monitor spec: selector: matchLabels: app: zookeeper endpoints: - port: metrics interval: 15s namespaceSelector: any: true

4. 自动证书轮换

bash

复制代码

# 证书更新后滚动重启
kubectl patch zk zookeeper-cluster --type='merge' -p '{"spec":{"tlsConfig":{"certUpdated":true}}}'

四、安全合规与灾备

1. 安全加固

yaml

复制代码

# 在CRD中增加安全配置
spec:
  security:
    enable: true jaasConfig: secretRef: zk-jaas-secret tlsConfig: enable: true secretRef: zk-tls-secret networkPolicy: enabled: true allowedClients: - 192.168.10.0/24

2. 跨集群灾备

yaml

复制代码

apiVersion: zookeeper.pravega.io/v1beta1
kind: ZookeeperCluster
metadata: name: zookeeper-dr spec: replicas: 3 config: # 配置为观察者模式 peerType: observer # 连接主集群 initConfig: | server.1=zk1:2888:3888:participant;2181 server.2=zk2:2888:3888:participant;2181 server.3=zk3:2888:3888:participant;2181 server.4=dr-zk1:2888:3888:observer;2181 server.5=dr-zk2:2888:3888:observer;2181 server.6=dr-zk3:2888:3888:observer;2181

五、日常运维操作

1. 集群状态检查

bash

复制代码

# 查看集群状态
kubectl get zookeepercluster
kubectl describe zk zookeeper-cluster

# 检查节点角色
kubectl exec zookeeper-cluster-0 -- zkServer.sh status

2. 日志管理

bash

复制代码

# 查看实时日志
kubectl logs -f zookeeper-cluster-0

# 日志归档配置 (Operator自动管理)

3. 配置热更新

bash

复制代码

# 修改配置后触发更新
kubectl patch zk zookeeper-cluster --type='merge' -p '{"spec":{"config":{"tickTime":"3000"}}}'

六、扩展与升级

1. 集群升级流程

bash

复制代码

# 滚动升级到新版本
kubectl patch zk zookeeper-cluster --type='merge' -p '{"spec":{"image":{"tag":"3.9.0"}}}' # 监控升级进度 kubectl get pods -w -l app=zookeeper

2. 多集群管理

bash

复制代码

# 部署多套Zookeeper集群
kubectl apply -f zookeeper-cluster-app1.yaml
kubectl apply -f zookeeper-cluster-app2.yaml

# 统一监控
kubectl apply -f zookeeper-global-monitor.yaml

七、备份与恢复方案

1. Velero 全集群备份

bash

复制代码

# 安装Velero
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.0.0 \ --bucket zk-backups \ --secret-file ./credentials-velero \ --use-restic # 创建备份 velero backup create zk-full-backup --include-namespaces default --selector app=zookeeper # 灾难恢复 velero restore create --from-backup zk-full-backup

2. 数据迁移

bash

复制代码

# 使用zkTransfer工具
kubectl exec zookeeper-cluster-0 -- zkTransfer.sh \
  --source zk1:2181 \
  --target new-zk1:2181 \ --path /critical_data \ --parallel 8

运维检查清单

检查项	频率	命令/方法
集群健康状态	每日	`kubectl get zk`
节点资源使用率	每日	`kubectl top pods`
证书有效期检查	每月	`keytool -list -v -keystore`
备份恢复测试	每季度	Velero恢复演练
安全漏洞扫描	每月	Trivy扫描镜像
故障转移演练	每半年	模拟节点故障
性能压测	每年	ZK Benchmark工具

通过Kubernetes Operator实现Zookeeper集群的全生命周期自动化管理，结合Velero实现灾备，Prometheus实现监控，显著提升运维效率。生产环境建议：

使用HashiCorp Vault管理密钥

部署多可用区集群

集成OpenPolicyAgent进行策略管理

使用GitOps工作流（Argo CD）管理配置