K8s集群运行时自动化运维全覆盖落地实操(上)

K8s集群运行时自动化运维全覆盖落地实操

|--------|-----------------------------------------------------------------------------------|
| 项目 | 内容 |
| 适用版本 | Kubernetes v1.28.15 + Ansible 2.15 + ArgoCD 2.9 + cert-manager 1.13 + Velero 1.12 |
| 核心目标 | 运维全流程自动化、事件驱动自愈、基础设施即代码、零人工干预 |
| 编写日期 | 2026-08-18 |
| 文档版本 | v1.0 |

1. 操作环境与前置准备

1.1 硬件要求

|--------|---------|--------|-----------|-------------------|
| 角色 | CPU | 内存 | 硬盘 | 说明 |
| Worker | 8核+ | 16GB+ | 200GB SSD | 业务节点 |
| 运维节点 | 8核+ | 16GB+ | 200GB SSD | Ansible/ArgoCD/CI |
| 备份存储 | - | - | 1TB+ | MinIO/S3对象存储 |

1.2 软件版本矩阵

|--------------------|---------|------------|
| 组件 | 版本 | 用途 |
| containerd | 1.7.11+ | 容器运行时 |
| Ansible | 2.15+ | 自动化部署与配置 |
| ArgoCD | 2.9.x | GitOps持续交付 |
| Helm | 3.13+ | K8s包管理 |
| cert-manager | 1.13.x | 自动化证书管理 |
| Velero | 1.12.x | 备份与恢复 |
| HPA | v1/v2 | 水平自动扩缩容 |
| VPA | 0.14+ | 垂直自动扩缩容 |
| KEDA | 2.12+ | 事件驱动扩缩容 |
| Trivy | 0.48+ | 镜像安全扫描 |
| kube-bench | 0.7+ | 安全审计 |
| Metrics Server | 0.7+ | 资源指标采集 |
| Prometheus Adapter | 0.11+ | 自定义指标 |
| MinIO | RELEASE | 对象存储（备份） |

1.3 节点规划

|-----------------|------------------|--------|-------------------|
| 主机名 | IP地址 | 角色 | 部署组件 |
| k8s-worker01-05 | 192.168.10.21-25 | Worker | 业务+自动化组件 |
| ops-node | 192.168.10.40 | 运维 | Ansible/CI Runner |
| minio | 192.168.10.50 | 存储 | Velero备份后端 |

1.4 自动化运维成熟度评估

|--------|--------|-------------------|---------|
| 等级 | 名称 | 特征 | 覆盖率 |
| L2 | 脚本化 | 常用操作脚本化，手动触发 | 10-30% |
| L3 | 工具化 | Ansible/CI工具，半自动化 | 30-60% |
| L4 | 平台化 | GitOps+自愈+自动扩缩容 | 60-90% |
| L5 | 智能化 | AIOps预测+自动决策+零干预 | >90% |

2. 自动化运维体系架构

2.1 自动化运维全景图

┌─────────────────────────────────────────────────────────────────┐
│ 自动化运维全景图 │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ 交付自动化 │ │ 运行自动化 │ │ 安全自动化 │ │
│ │ Ansible │ │ HPA/VPA │ │ Trivy │ │
│ │ ArgoCD │ │ CA │ │ kube-bench │ │
│ │ Helm │ │ 自愈脚本 │ │ 网络策略 │ │
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
│ │ │ │ │
│ └────────────────┼────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────┐ │
│ │ 事件驱动引擎 │ │
│ │ Prometheus告警 → Webhook → 自动处理脚本/Operator │ │
│ └──────────────────────┬──────────────────────────┘ │
│ │ │
│ ┌──────────────────────▼──────────────────────────┐ │
│ │ ChatOps协作层 │ │
│ │ 钉钉/企微机器人 → 告警通知 → 交互命令 → 审批流 │ │
│ └──────────────────────┬──────────────────────────┘ │
│ │ │
│ ┌──────────────────────▼──────────────────────────┐ │
│ │ 运维平台门户 │ │
│ │ 作业模板 │ 审批流 │ 操作审计 │ 效率度量 │ │
│ └─────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘

2.2 自动化分层模型

|--------|---------------|-------------------------|----------|
| 层级 | 自动化内容 | 工具 | 触发方式 |
| 集群层 | K8s部署、升级、节点管理 | Ansible/kubeadm | 手动/事件 |
| 应用层 | 部署、回滚、配置 | ArgoCD/Helm | Git推送自动 |
| 运行层 | 扩缩容、自愈、备份 | HPA/VPA/Velero/Operator | 自动/定时 |
| 安全层 | 扫描、审计、证书 | Trivy/cert-manager | 自动/定时 |
| 协作层 | 通知、审批、工单 | ChatOps/Webhook | 事件驱动 |

2.3 工具链选型

|--------|--------------|---------------|---------------|
| 领域 | 选型 | 替代方案 | 选型理由 |
| 持续交付 | ArgoCD | FluxCD | UI友好、多集群、企业级 |
| 包管理 | Helm | Kustomize | 模板化、生态丰富 |
| 扩缩容 | HPA+KEDA | 自定义Controller | 官方标准+事件驱动扩展 |
| 备份 | Velero | Stash | 官方推荐、功能全面 |
| 证书 | cert-manager | 手动管理 | 自动化续期、多Issuer |
| 安全扫描 | Trivy | Clair/Anchore | 速度快、集成简单 |
| 对象存储 | MinIO | AWS S3/OSS | 私有化、S3兼容 |

3. 基础设施即代码（IaC）

3.1 Ansible自动化部署

安装Ansible：

运维节点安装Ansible

yum install -y python3-pip
pip3 install ansible==2.15.0
ansible --version

安装K8s相关集合

ansible-galaxy collection install community.kubernetes
ansible-galaxy collection install ansible.posix

创建Inventory：

/opt/ansible/inventory/prod.ini

masters

k8s-master01 ansible_host=192.168.10.10
k8s-master02 ansible_host=192.168.10.11
k8s-master03 ansible_host=192.168.10.12

workers

k8s-worker01 ansible_host=192.168.10.21
k8s-worker02 ansible_host=192.168.10.22
k8s-worker03 ansible_host=192.168.10.23
k8s-worker04 ansible_host=192.168.10.24
k8s-worker05 ansible_host=192.168.10.25

k8s:children

masters
workers

k8s:vars

ansible_user=root
ansible_ssh_private_key_file=/root/.ssh/id_rsa
k8s_version=1.28.15
containerd_version=1.7.11
pod_cidr=10.244.0.0/16
service_cidr=10.96.0.0/12

3.2 集群初始化Playbook

创建 01-system-init.yml：

---

• name: K8s节点系统初始化
  hosts: k8s
  become: yes
  tasks:

• name: 设置主机名
  hostname:
  name: "{{ inventory_hostname }}"

• name: 配置hosts
  lineinfile:
  path: /etc/hosts
  line: "{{ hostvarsitem.ansible_host }} {{ item }}"
  loop: "{{ groups'k8s' }}"

• name: 关闭防火墙
  systemd:
  name: firewalld
  state: stopped
  enabled: no

• name: 关闭SELinux
  selinux:
  state: disabled

• name: 关闭Swap
  command: swapoff -a
  changed_when: false

• name: 注释fstab swap
  replace:
  path: /etc/fstab
  regexp: '^(\^#.*swap.*)$'
  replace: '#\1'

• name: 加载内核模块
  modprobe:
  name: "{{ item }}"
  loop:

• overlay

• br_netfilter

• ip_vs

• ip_vs_rr

• ip_vs_wrr

• ip_vs_sh

• nf_conntrack

• name: 配置模块开机加载
  copy:
  content: |
  overlay
  br_netfilter
  ip_vs
  ip_vs_rr
  ip_vs_wrr
  ip_vs_sh
  nf_conntrack
  dest: /etc/modules-load.d/k8s.conf

• name: 配置内核参数
  copy:
  content: |
  net.bridge.bridge-nf-call-iptables = 1
  net.bridge.bridge-nf-call-ip6tables = 1
  net.ipv4.ip_forward = 1
  net.ipv6.conf.all.disable_ipv6 = 1
  vm.swappiness = 0
  net.netfilter.nf_conntrack_max = 1048576
  dest: /etc/sysctl.d/99-k8s.conf
  notify: reload sysctl

• name: 配置时间同步
  yum:
  name: chrony
  state: present

• systemd:
  name: chronyd
  state: started
  enabled: yes

handlers:

• name: reload sysctl
  command: sysctl --system

创建 02-containerd-install.yml：

---

• name: 安装containerd
  hosts: k8s
  become: yes
  tasks:

• name: 安装containerd
  yum:
  name: "containerd-{{ containerd_version }}"
  state: present

• name: 生成默认配置
  command: containerd config default
  register: containerd_config
  changed_when: false

• name: 写入配置文件
  copy:
  content: "{{ containerd_config.stdout }}"
  dest: /etc/containerd/config.toml

• name: 设置SystemdCgroup
  replace:
  path: /etc/containerd/config.toml
  regexp: 'SystemdCgroup = false'
  replace: 'SystemdCgroup = true'

• name: 设置pause镜像
  replace:
  path: /etc/containerd/config.toml
  regexp: 'sandbox_image = ".*"'
  replace: 'sandbox_image = "harbor.local/k8s/pause:3.9"'

• name: 启动containerd
  systemd:
  name: containerd
  state: started
  enabled: yes

创建 03-k8s-install.yml：

---

• name: 安装K8s组件
  hosts: k8s
  become: yes
  tasks:

• name: 安装kubelet kubeadm kubectl
  yum:
  name:

• "kubelet-{{ k8s_version }}"

• "kubeadm-{{ k8s_version }}"

• "kubectl-{{ k8s_version }}"
  state: present

• name: 配置kubelet
  copy:
  content: |
  KUBELET_EXTRA_ARGS=--cgroup-driver=systemd --container-runtime-endpoint=unix:///run/containerd/containerd.sock
  dest: /etc/sysconfig/kubelet

• name: 启动kubelet
  systemd:
  name: kubelet
  state: started
  enabled: yes

执行部署：

一键部署

cd /opt/ansible
ansible-playbook -i inventory/prod.ini 01-system-init.yml
ansible-playbook -i inventory/prod.ini 02-containerd-install.yml
ansible-playbook -i inventory/prod.ini 03-k8s-install.yml

3.3 配置管理与幂等性

Ansible幂等性设计原则：

幂等性示例：只在配置变化时触发重启

• name: 修改containerd配置
  template:
  src: config.toml.j2
  dest: /etc/containerd/config.toml
  notify: restart containerd

handlers:

• name: restart containerd
  systemd:
  name: containerd
  state: restarted

配置模板化（Jinja2）：

/opt/ansible/templates/config.toml.j2

version = 2
root = "{{ containerd_data_dir | default('/data/containerd') }}"
state = "/run/containerd"

plugins."io.containerd.grpc.v1.cri"

sandbox_image = "{{ pause_image }}"

plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc

plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options

SystemdCgroup = {{ systemd_cgroup | default(true) | lower }}

3.4 环境一致性保障

环境差异检查

ansible -i inventory/prod.ini k8s -m shell -a "containerd --version; kubelet --version; uname -r"

配置漂移检测

ansible -i inventory/prod.ini k8s -m copy -a "
src=/opt/ansible/templates/config.toml.j2
dest=/etc/containerd/config.toml
checksum=yes
" --check

批量配置对齐（dry-run）

ansible-playbook -i inventory/prod.ini 02-containerd-install.yml --check --diff

4. GitOps持续交付

4.1 ArgoCD部署与配置

创建命名空间

kubectl create namespace argocd

安装ArgoCD

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.9.3/manifests/install.yaml

或离线安装

kubectl apply -n argocd -f /opt/argocd/install.yaml

暴露服务（NodePort）

kubectl patch svc argocd-server -n argocd -p '{"spec":{"type":"NodePort"}}'

获取初始密码

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

登录

argocd login <argocd-url> --username admin --password <password>

配置Git仓库

argocd repo add https://gitlab.company.com/k8s/manifests.git \
--username gitops --password <token>

4.2 应用声明式管理

创建Application：

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: web-app
namespace: argocd
spec:
project: default
source:
repoURL: https://gitlab.company.com/k8s/manifests.git
targetRevision: main
path: apps/web-app/overlays/prod
destination:
server: https://kubernetes.default.svc
namespace: prod
syncPolicy:
automated:
prune: true
selfHeal: true
allowEmpty: false
syncOptions:

• CreateNamespace=true
• PrunePropagationPolicy=foreground
  ignoreDifferences:
• group: apps
  kind: Deployment
  jsonPointers:
• /spec/replicas

Helm应用：

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: monitoring
namespace: argocd
spec:
project: default
source:
repoURL: https://prometheus-community.github.io/helm-charts
chart: kube-prometheus-stack
targetRevision: 55.0.0
helm:
releaseName: monitoring
valueFiles:

• values-prod.yaml
  destination:
  server: https://kubernetes.default.svc
  namespace: monitoring
  syncPolicy:
  automated:
  prune: true
  selfHeal: true

4.3 多环境管理

目录结构：

apps/
└── web-app/
├── base/
│ ├── deployment.yaml
│ ├── service.yaml
│ └── kustomization.yaml
└── overlays/
├── dev/
│ ├── kustomization.yaml
│ └── replicas-patch.yaml
├── staging/
│ ├── kustomization.yaml
│ └── replicas-patch.yaml
└── prod/
├── kustomization.yaml
└── replicas-patch.yaml

Kustomize覆盖：

overlays/prod/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:

• ../../base
  namespace: prod
  patches:
• path: replicas-patch.yaml
  images:
• name: web-app
  newTag: v1.2.3

多环境ApplicationSet：

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: web-app-multi-env
spec:
generators:

• list:
  elements:
• env: dev
  namespace: dev
  replicas: "1"
• env: staging
  namespace: staging
  replicas: "2"
• env: prod
  namespace: prod
  replicas: "5"
  template:
  metadata:
  name: "web-app-{{env}}"
  spec:
  project: default
  source:
  repoURL: https://gitlab.company.com/k8s/manifests.git
  targetRevision: main
  path: "apps/web-app/overlays/{{env}}"
  destination:
  server: https://kubernetes.default.svc
  namespace: "{{namespace}}"
  syncPolicy:
  automated:
  prune: true
  selfHeal: true

4.4 自动同步与回滚

查看应用同步状态

argocd app list
argocd app get web-app

手动同步

argocd app sync web-app

查看历史版本

argocd app history web-app

回滚到上一版本

argocd app rollback web-app HEAD~1

回滚到指定版本

argocd app rollback web-app 5

暂停自动同步（紧急情况）

argocd app set web-app --sync-policy none

恢复自动同步

argocd app set web-app --sync-policy automated

5. 自动化扩缩容

5.1 HPA水平自动扩缩容

部署Metrics Server：

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

离线环境需修改镜像地址

kubectl edit deployment metrics-server -n kube-system

添加 --kubelet-insecure-tls 参数

创建HPA：

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: web-app-hpa
namespace: prod
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: web-app
minReplicas: 3
maxReplicas: 20
metrics:

• type: Resource
  resource:
  name: cpu
  target:
  type: Utilization
  averageUtilization: 70
• type: Resource
  resource:
  name: memory
  target:
  type: Utilization
  averageUtilization: 80
  behavior:
  scaleUp:
  stabilizationWindowSeconds: 60
  policies:
• type: Percent
  value: 100
  periodSeconds: 60
• type: Pods
  value: 4
  periodSeconds: 60
  selectPolicy: Max
  scaleDown:
  stabilizationWindowSeconds: 300
  policies:
• type: Percent
  value: 25
  periodSeconds: 60

验证：

kubectl get hpa -n prod
kubectl describe hpa web-app-hpa -n prod
kubectl get hpa web-app-hpa -n prod -w

5.2 VPA垂直自动扩缩容

安装VPA

git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler
./hack/vpa-up.sh

验证

kubectl get pods -n kube-system | grep vpa

创建VPA：

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: web-app-vpa
namespace: prod
spec:
targetRef:
apiVersion: "apps/v1"
kind: Deployment
name: web-app
updatePolicy:
updateMode: "Auto" # Off/Initial/Auto/Recreate
resourcePolicy:
containerPolicies:

• containerName: '*'
  minAllowed:
  cpu: 100m
  memory: 128Mi
  maxAllowed:
  cpu: "4"
  memory: 8Gi
  controlledResources: "cpu", "memory"

**注意**：HPA和VPA不能同时用于同一资源指标（CPU/内存），可组合使用（HPA基于自定义指标，VPA调整CPU/内存）。

5.3 Cluster Autoscaler节点自动伸缩

cluster-autoscaler部署（以云厂商为例，私有化需结合云API）

apiVersion: apps/v1
kind: Deployment
metadata:
name: cluster-autoscaler
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: cluster-autoscaler
template:
metadata:
labels:
app: cluster-autoscaler
spec:
containers:

• name: cluster-autoscaler
  image: harbor.local/k8s/cluster-autoscaler:v1.28.0
  command:
• ./cluster-autoscaler
• --cloud-provider=xxx # 云厂商类型
• --nodes=3:10:worker-pool
• --scale-down-delay-after-add=10m
• --scale-down-unneeded-time=10m
• --scale-down-utilization-threshold=0.5
• --skip-nodes-with-local-storage=false
• --skip-nodes-with-system-pods=false

私有化环境节点自动扩缩容方案：

方案1：结合虚拟化平台API（Proxmox/VMware）自动创建虚拟机

方案2：预部署空闲节点池，通过节点标签激活/休眠

方案3：使用KubeKey/k3s等轻量方案快速加入节点

节点池预热脚本

#!/bin/bash

node-pool-autoscaler.sh

PENDING_PODS=(kubectl get pods -A --field-selector=status.phase=Pending \| grep -v NAME \| wc -l) IDLE_NODES=(kubectl get nodes -l pool=idle --no-headers | grep Ready | wc -l)

if ${PENDING_PODS} -gt 0 && ${IDLE_NODES} -gt 0 ; then
echo "有{PENDING_PODS}个Pending Pod，激活空闲节点..." IDLE_NODE=(kubectl get nodes -l pool=idle --no-headers | grep Ready | head -1 | awk '{print 1}') kubectl label node {IDLE_NODE} pool-
kubectl uncordon ${IDLE_NODE}
fi

5.4 自定义指标扩缩容

部署Prometheus Adapter：

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus-adapter prometheus-community/prometheus-adapter \
--namespace monitoring \
--set prometheus.url=http://prometheus.monitoring.svc \
--set rules.default=true

自定义指标HPA：

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: api-gateway-hpa
namespace: prod
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: api-gateway
minReplicas: 2
maxReplicas: 50
metrics:

• type: Pods
  pods:
  metric:
  name: http_requests_per_second
  target:
  type: AverageValue
  averageValue: "100"
• type: Object
  object:
  metric:
  name: nginx_ingress_controller_requests
  describedObject:
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  name: api-gateway
  target:
  type: AverageValue
  averageValue: "500"

5.5 定时扩缩容（KEDA）

安装KEDA

helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda --namespace keda --create-namespace

基于Cron定时扩缩容：

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: web-app-cron-scaler
namespace: prod
spec:
scaleTargetRef:
name: web-app
minReplicaCount: 2
maxReplicaCount: 20
triggers:

• type: cron
  metadata:
  timezone: Asia/Shanghai
  start: "0 9 * * *" # 每天9点扩容
  end: "0 22 * * *" # 每天22点缩容
  desiredReplicas: "15"
• type: cpu
  metricType: Utilization
  metadata:
  value: "70"

基于消息队列长度扩缩容：

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: consumer-scaler
spec:
scaleTargetRef:
name: message-consumer
minReplicaCount: 1
maxReplicaCount: 50
triggers:

• type: rabbitmq
  metadata:
  host: amqp://user:pass@rabbitmq.prod.svc:5672/
  queueName: task_queue
  queueLength: "100"

6. 自动化备份与恢复

6.1 Velero部署

部署MinIO（备份后端）：

安装MinIO Operator或单机版

docker run -d \
--name minio \
-p 9000:9000 -p 9001:9001 \
-e MINIO_ROOT_USER=minioadmin \
-e MINIO_ROOT_PASSWORD=minioadmin \
-v /data/minio:/data \
minio/minio server /data --console-address ":9001"

创建bucket

mc alias set local http://192.168.10.50:9000 minioadmin minioadmin
mc mb local/k8s-backup

安装Velero：

下载Velero CLI

wget https://github.com/vmware-tanzu/velero/releases/download/v1.12.2/velero-v1.12.2-linux-amd64.tar.gz
tar xzf velero-v1.12.2-linux-amd64.tar.gz
install -m 755 velero-v1.12.2-linux-amd64/velero /usr/local/bin/

创建凭证文件

cat > /tmp/credentials-velero << 'EOF'

default

aws_access_key_id = minioadmin
aws_secret_access_key = minioadmin
EOF

安装Velero

velero install \
--provider aws \
--plugins harbor.local/k8s/velero-plugin-for-aws:v1.8.0 \
--bucket k8s-backup \
--backup-location-config region=minio,s3ForcePathStyle=true,s3Url=http://192.168.10.50:9000 \
--snapshot-location-config region=minio \
--secret-file /tmp/credentials-velero \
--use-volume-snapshots=false \
--use-restic

验证

kubectl get pods -n velero
velero version

6.2 资源自动备份策略

创建定时备份：

apiVersion: velero.io/v1
kind: Schedule
metadata:
name: daily-full-backup
namespace: velero
spec:
schedule: "0 2 * * *" # 每天凌晨2点
template:
includedNamespaces:

• '*'
  excludedNamespaces:
• kube-system
• velero
  ttl: 720h # 保留30天
  storageLocation: default
  hooks: {}

---

apiVersion: velero.io/v1
kind: Schedule
metadata:
name: hourly-critical-backup
namespace: velero
spec:
schedule: "0 * * * *" # 每小时
template:
includedNamespaces:

• prod
  labelSelector:
  matchLabels:
  backup: critical
  ttl: 72h # 保留3天
  includeClusterResources: true

---

apiVersion: velero.io/v1
kind: Schedule
metadata:
name: weekly-etcd-backup
namespace: velero
spec:
schedule: "0 3 * * 0" # 每周日凌晨3点
template:
includedResources:

• pods
• deployments
• services
• configmaps
• secrets
• persistentvolumeclaims
  ttl: 720h

手动备份：

全量备份

velero backup create full-backup-$(date +%Y%m%d) --wait

按命名空间备份

velero backup create prod-backup --include-namespaces prod --wait

按标签备份

velero backup create critical-backup --selector backup=critical --wait

查看备份列表

velero backup get

查看备份详情

velero backup describe full-backup-20260818
velero backup logs full-backup-20260818

6.3 持久卷快照备份

使用restic备份PVC数据：

启用restic

velero restic repo create --include-namespaces prod

备份时包含PVC

velero backup create with-pvc --include-namespaces prod --wait

Pod注解启用restic备份

spec.template.metadata.annotations:

backup.velero.io/backup-volumes: data,logs

apiVersion: apps/v1
kind: Deployment
metadata:
name: database
namespace: prod
spec:
template:
metadata:
annotations:
backup.velero.io/backup-volumes: db-data
spec:
containers:

• name: mysql
  volumeMounts:
• name: db-data
  mountPath: /var/lib/mysql
  volumes:
• name: db-data
  persistentVolumeClaim:
  claimName: mysql-pvc

6.4 自动化恢复演练

恢复脚本：

#!/bin/bash

restore-drill.sh

定期恢复演练（建议每月一次）

BACKUP_NAME=1 \[ -z "BACKUP_NAME" ] && BACKUP_NAME=$(velero backup get -o jsonpath='{.items-1.metadata.name}')

echo "=== 开始恢复演练 ==="
echo "使用备份: ${BACKUP_NAME}"

1. 创建测试命名空间

kubectl create namespace restore-test

2. 恢复到测试命名空间

velero restore create restore-drill-(date +%Y%m%d) \\ --from-backup {BACKUP_NAME} \
--namespace-mappings prod:restore-test \
--wait

3. 验证恢复结果

echo "恢复的资源:"
kubectl get all -n restore-test

4. 验证数据完整性

kubectl exec -n restore-test deploy/database -- mysql -e "SELECT COUNT(*) FROM users;"

5. 清理测试环境

velero restore delete restore-drill-$(date +%Y%m%d) --confirm
kubectl delete namespace restore-test

echo "=== 恢复演练完成 ==="

定时恢复演练（CronJob）：

apiVersion: batch/v1
kind: CronJob
metadata:
name: monthly-restore-drill
namespace: velero
spec:
schedule: "0 4 1 * *" # 每月1号凌晨4点
jobTemplate:
spec:
template:
spec:
containers:

• name: restore-drill
  image: harbor.local/k8s/velero:v1.12.2
  command:
• /bin/sh
• -c
• |

恢复演练逻辑

velero restore create drill-$(date +%Y%m%d) --from-schedule daily-full-backup --namespace-mappings prod:drill-test --wait

验证...

清理...

restartPolicy: OnFailure

7. 自动化证书管理

7.1 cert-manager部署

安装cert-manager

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.3/cert-manager.yaml

离线安装

kubectl apply -f /opt/cert-manager/cert-manager.yaml

验证

kubectl get pods -n cert-manager
kubectl get crd | grep cert-manager

7.2 Let's Encrypt自动签发

创建ClusterIssuer：

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: admin@company.com
privateKeySecretRef:
name: letsencrypt-prod
solvers:

• http01:
  ingress:
  class: nginx
  selector:
  dnsZones:
• "company.com"

---

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
email: admin@company.com
privateKeySecretRef:
name: letsencrypt-staging
solvers:

• http01:
  ingress:
  class: nginx

Ingress自动签发证书：

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: web-app
namespace: prod
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-prod"
cert-manager.io/renew-before: "720h" # 到期前30天续期
spec:
tls:

• hosts:
• app.company.com
  secretName: web-app-tls
  rules:
• host: app.company.com
  http:
  paths:
• path: /
  pathType: Prefix
  backend:
  service:
  name: web-app
  port:
  number: 80

7.3 内部CA证书管理

创建内部CA：

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: internal-ca
spec:
ca:
secretName: internal-ca-secret

生成CA证书（一次性）

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: internal-ca
namespace: cert-manager
spec:
isCA: true
commonName: internal-ca
secretName: internal-ca-secret
issuerRef:
name: selfsigned
kind: ClusterIssuer

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: selfsigned
spec:
selfSigned: {}

内部服务证书：

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: database-tls
namespace: prod
spec:
secretName: database-tls
duration: 8760h # 1年
renewBefore: 720h # 到期前30天
issuerRef:
name: internal-ca
kind: ClusterIssuer
commonName: database.prod.svc.cluster.local
dnsNames:

• database
• database.prod
• database.prod.svc
• database.prod.svc.cluster.local
  ipAddresses:
• 10.96.0.100
  usages:
• server auth
• client auth

7.4 证书自动续期与告警

查看证书状态

kubectl get certificates -A
kubectl describe certificate web-app-tls -n prod

手动触发续期

kubectl cert-manager renew web-app-tls -n prod

证书到期监控（Prometheus）

cert-manager内置指标：certmanager_certificate_expiration_timestamp_seconds

证书到期告警规则：

groups:

• name: cert-manager
  rules:

• alert: CertificateExpiringSoon
  expr: |
  (certmanager_certificate_expiration_timestamp_seconds - time()) < 86400 * 30
  for: 1h
  labels:
  severity: warning
  annotations:
  summary: "证书30天内到期"
  description: "证书{{ labels.name }}在{{ value | humanizeDuration }}后到期"

• alert: CertificateExpiringCritical
  expr: |
  (certmanager_certificate_expiration_timestamp_seconds - time()) < 86400 * 7
  for: 0m
  labels:
  severity: critical
  annotations:
  summary: "证书7天内到期"