K8s集群运行时自动化运维全覆盖落地实操(下)

8. 自动化安全运维

8.1 镜像安全扫描（Trivy）

CI流水线集成扫描：

.gitlab-ci.yml

stages:

• build

• scan

• deploy

scan_image:

stage: scan

image: harbor.local/k8s/trivy:0.48.0

script:

扫描镜像，高危漏洞阻断部署

• trivy image --exit-code 1 --severity HIGH,CRITICAL --no-progress IMAGE_NAME:CI_COMMIT_SHORT_SHA

生成SBOM

• trivy image --format spdx-json --output sbom.json IMAGE_NAME:CI_COMMIT_SHORT_SHA

artifacts:

reports:

cyclonedx: sbom.json

集群内运行时扫描（Operator）：

安装Trivy Operator

helm repo add aqua https://aquasecurity.github.io/helm-charts/

helm install trivy-operator aqua/trivy-operator \

--namespace trivy-system \

--create-namespace \

--set trivy.ignoreUnfixed=true \

--set scanJobs.concurrentScanJobsLimit=3

扫描结果查看：

kubectl get vulnerabilityreports -A

kubectl describe vulnerabilityreports -n prod <pod-name>

8.2 配置安全审计（kube-bench）

运行kube-bench

kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml

查看结果

kubectl logs -n kube-system job.batch/kube-bench

定时审计CronJob：

apiVersion: batch/v1

kind: CronJob

metadata:

name: kube-bench-audit

namespace: kube-system

spec:

schedule: "0 6 * * 1" # 每周一早上6点

jobTemplate:

spec:

template:

spec:

hostPID: true

containers:

• name: kube-bench

image: harbor.local/k8s/kube-bench:0.7.2

command: "kube-bench", "--json"

volumeMounts:

• name: var-lib-etcd

mountPath: /var/lib/etcd

• name: etc-kubernetes

mountPath: /etc/kubernetes

restartPolicy: Never

volumes:

• name: var-lib-etcd

hostPath:

path: /var/lib/etcd

• name: etc-kubernetes

hostPath:

path: /etc/kubernetes

8.3 网络策略自动化

默认拒绝策略（所有命名空间）：

通过Kyverno或ArgoCD自动应用到所有命名空间

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

name: default-deny

spec:

podSelector: {}

policyTypes:

• Ingress

• Egress

egress:

• to:

• namespaceSelector: {}

ports:

• protocol: UDP

port: 53

• protocol: TCP

port: 53

Kyverno自动注入网络策略：

apiVersion: kyverno.io/v1

kind: ClusterPolicy

metadata:

name: inject-default-networkpolicy

spec:

rules:

• name: add-default-deny

match:

any:

• resources:

kinds:

• Namespace

generate:

kind: NetworkPolicy

name: default-deny

namespace: "{{request.object.metadata.name}}"

data:

spec:

podSelector: {}

policyTypes:

• Ingress

• Egress

8.4 RBAC权限自动化管理

命名空间权限模板：

通过ArgoCD ApplicationSet为每个项目自动创建RBAC

apiVersion: argoproj.io/v1alpha1

kind: ApplicationSet

metadata:

name: project-rbac

spec:

generators:

• git:

repoURL: https://gitlab.company.com/k8s/projects.git

files:

• path: "projects/*.yaml"

template:

metadata:

name: "rbac-{{name}}"

spec:

project: default

source:

repoURL: https://gitlab.company.com/k8s/rbac-templates.git

path: base

helm:

parameters:

• name: namespace

value: "{{name}}"

• name: team

value: "{{owner}}"

destination:

server: https://kubernetes.default.svc

namespace: "{{name}}"

8.5 漏洞自动修复

#!/bin/bash

auto-patch-images.sh

自动重建包含高危漏洞的镜像

获取有高危漏洞的镜像列表

VULN_IMAGES=$(kubectl get vulnerabilityreports -A -o json | \

jq -r '.items\[\] | select(.report.summary.criticalCount > 0) | .report.artifact.repository + ":" + .report.artifact.tag' | sort -u)

for image in $VULN_IMAGES; do

echo "发现高危漏洞镜像: $image"

触发CI重新构建（拉取最新基础镜像）

curl -X POST "https://gitlab.company.com/api/v4/projects/.../trigger/pipeline" ...

done

9. 事件驱动自动化与自愈

9.1 节点自动修复

节点健康检查脚本：

#!/bin/bash

node-auto-repair.sh

由Alertmanager Webhook触发

NODE=$1

ACTION=$2

log() { echo "$(date) $1" >> /var/log/node-repair.log; }

log "节点{NODE}触发自动修复: {ACTION}"

case ${ACTION} in

"notready")

尝试重启kubelet和containerd

ssh root@${NODE} "systemctl restart containerd kubelet"

sleep 30

STATUS=(kubectl get node {NODE} -o jsonpath='{.status.conditions?(@.type=="Ready").status}')

if "${STATUS}" != "True" ; then

log "重启服务无效，尝试重启节点..."

ssh root@${NODE} "reboot"

fi

;;

"disk-pressure")

ssh root@${NODE} "

crictl rmi --prune

journalctl --vacuum-size=500M

find /tmp -type f -atime +7 -delete

find /var/log -name '*.log' -mtime +7 -delete

"

;;

"memory-pressure")

驱逐低优先级Pod

kubectl drain ${NODE} --ignore-daemonsets --delete-emptydir-data --force --timeout=300s

;;

*)

log "未知操作: ${ACTION}"

;;

esac

Alertmanager触发自愈：

Alertmanager配置中添加webhook接收器

receivers:

• name: 'node-repair-webhook'

webhook_configs:

• url: 'http://ops-node:8080/auto-repair'

send_resolved: false

9.2 Pod异常自动处理

#!/bin/bash

pod-auto-heal.sh

定时检查并处理异常Pod

CrashLoopBackOff Pod自动重建

CRASHPODS=$(kubectl get pods -A -o json | \

jq -r '.items\[\] | select(.status.containerStatuses\[\].state.waiting.reason=="CrashLoopBackOff") | .metadata.namespace + "/" + .metadata.name')

for pod in $CRASHPODS; do

NS=(echo pod | cut -d/ -f1)

NAME=(echo pod | cut -d/ -f2)

RESTARTS=(kubectl get pod NAME -n $NS -o jsonpath='{.status.containerStatuses0.restartCount}')

if "$RESTARTS" -gt 10 ; then

echo "Pod pod重启{RESTARTS}次，执行删除重建..."

kubectl delete pod NAME -n NS

fi

done

Pending超过30分钟的Pod告警

PENDING=(kubectl get pods -A --field-selector=status.phase=Pending --no-headers \| awk '6 > "30m" {print}')

-n "$PENDING" && echo "⚠ 存在长时间Pending Pod:" && echo "$PENDING"

9.3 磁盘自动清理

#!/bin/bash

disk-auto-cleanup.sh

所有节点定时执行（每小时）

THRESHOLD=85

USAGE=(df / \| awk 'NR==2{print 5}' | tr -d '%')

if ${USAGE} -gt ${THRESHOLD} ; then

echo "磁盘使用率${USAGE}%，执行清理..."

1. 清理containerd未使用镜像

crictl rmi --prune

2. 清理已退出容器

crictl rm (crictl ps -a \| grep Exited \| awk '{print 1}') 2>/dev/null

3. 清理journal日志

journalctl --vacuum-size=200M

4. 清理/tmp旧文件

find /tmp -type f -atime +3 -delete

5. 清理旧日志

find /var/log -name '*.log' -mtime +7 -exec truncate -s 0 {} \;

6. 清理containerd旧数据

ctr -n k8s.io content prune 2>/dev/null

echo "清理完成，当前使用率: (df / \| awk 'NR==2{print 5}')"

fi

CronJob部署：

apiVersion: batch/v1

kind: CronJob

metadata:

name: disk-cleanup

namespace: kube-system

spec:

schedule: "0 * * * *" # 每小时

jobTemplate:

spec:

template:

spec:

hostNetwork: true

hostPID: true

containers:

• name: cleanup

image: harbor.local/k8s/busybox:1.36

command: "/bin/sh", "-c", "/scripts/cleanup.sh"

volumeMounts:

• name: scripts

mountPath: /scripts

• name: docker-sock

mountPath: /var/run

restartPolicy: OnFailure

volumes:

• name: scripts

configMap:

name: disk-cleanup-script

• name: docker-sock

hostPath:

path: /var/run

9.4 自定义Operator开发

使用Operator SDK创建自定义控制器：

安装Operator SDK

export ARCH=(case (uname -m) in x86_64) echo -n amd64 ;; aarch64) echo -n arm64 ;; esac)

export OS=(uname \| awk '{print tolower(0)}')

curl -LO https://github.com/operator-framework/operator-sdk/releases/latest/download/operator-sdk_${OS}_${ARCH}

install -m 755 operator-sdk_{OS}_{ARCH} /usr/local/bin/operator-sdk

创建项目

operator-sdk init --domain company.com --repo gitlab.company.com/k8s/app-operator

operator-sdk create api --group app --version v1 --kind WebApp --resource --controller

自定义Operator核心逻辑：

// controllers/webapp_controller.go

func (r *WebAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {

// 获取WebApp资源

var webApp appv1.WebApp

if err := r.Get(ctx, req.NamespacedName, &webApp); err != nil {

return ctrl.Result{}, client.IgnoreNotFound(err)

}

// 自动创建Deployment

dep := r.createDeployment(&webApp)

if err := r.CreateOrUpdate(ctx, dep); err != nil {

return ctrl.Result{}, err

}

// 自动创建Service

svc := r.createService(&webApp)

if err := r.CreateOrUpdate(ctx, svc); err != nil {

return ctrl.Result{}, err

}

// 自动创建HPA

if webApp.Spec.AutoScaling.Enabled {

hpa := r.createHPA(&webApp)

if err := r.CreateOrUpdate(ctx, hpa); err != nil {

return ctrl.Result{}, err

}

}

// 自动创建证书

if webApp.Spec.TLS.Enabled {

cert := r.createCertificate(&webApp)

if err := r.CreateOrUpdate(ctx, cert); err != nil {

return ctrl.Result{}, err

}

}

// 健康检查与自愈

if err := r.ensureHealth(ctx, &webApp); err != nil {

return ctrl.Result{}, err

}

return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil

}

10. ChatOps与协作自动化

10.1 钉钉/企微机器人集成

Webhook服务：

#!/usr/bin/env python3

chatops-server.py

from flask import Flask, request

import subprocess

import json

app = Flask(name)

def send_dingtalk(text):

import requests

webhook = "https://oapi.dingtalk.com/robot/send?access_token=xxx"

requests.post(webhook, json={"msgtype": "text", "text": {"content": text}})

@app.route('/alert', methods='POST')

def alert():

data = request.json

for alert in data.get('alerts', \[\]):

if alert'status' == 'firing':

msg = f"""🔴 告警触发

名称: {alert'labels'.get('alertname')}

级别: {alert'labels'.get('severity')}

节点: {alert'labels'.get('instance', 'N/A')}

描述: {alert'annotations'.get('description', 'N/A')}"""

send_dingtalk(msg)

return "ok"

@app.route('/command', methods='POST')

def command():

"""钉钉机器人交互命令"""

data = request.json

text = data.get('text', '').strip()

sender = data.get('senderNick', 'unknown')

if text.startswith('kubectl'):

result = subprocess.run(text, shell=True, capture_output=True, text=True, timeout=30)

send_dingtalk(f"执行结果:\n{result.stdout:2000}\n{result.stderr:500}")

elif text == '集群状态':

result = subprocess.run("kubectl get nodes -o wide", shell=True, capture_output=True, text=True)

send_dingtalk(result.stdout)

elif text == '异常Pod':

result = subprocess.run("kubectl get pods -A | grep -vE 'Running|Completed'", shell=True, capture_output=True, text=True)

send_dingtalk(result.stdout or "无异常Pod")

return "ok"

if name == 'main':

app.run(host='0.0.0.0', port=8080)

10.2 告警自动工单

告警自动创建工单

def create_ticket(alert):

severity = alert'labels'.get('severity')

title = f"{severity.upper()} {alert'labels'.get('alertname')}"

desc = alert'annotations'.get('description', '')

调用工单系统API

如Jira/禅道/自研工单系统

ticket_data = {

"title": title,

"description": desc,

"priority": "P0" if severity == "critical" else "P1",

"assignee": "sre-oncall",

"labels": "k8s", "automated"

}

response = requests.post("https://ticket.company.com/api", json=ticket_data)

print(f"创建工单: {title}")

10.3 交互式运维命令

钉钉机器人支持的命令：

|-----------------------|-----------|--------|
| 命令 | 功能 | 权限 |
| 异常Pod | 查看异常Pod列表 | 全员 |
| kubectl get xxx | 执行只读命令 | SRE |
| 重启Pod <ns> <name> | 重启指定Pod | SRE |
| 扩容 <deploy> <副本数> | 手动扩容 | SRE+审批 |
| 备份 <ns> | 触发即时备份 | SRE |
| 升级状态 | 查看升级进度 | 全员 |

10.4 变更自动通知

#!/bin/bash

notify-change.sh

ArgoCD同步Webhook触发

APP=$1

STATUS=$2

ENV=$3

if "$STATUS" = "Synced" ; then

MSG="✅ 应用{APP}({ENV})部署成功"

elif "$STATUS" = "OutOfSync" ; then

MSG="⚠️ 应用{APP}({ENV})配置漂移"

elif "$STATUS" = "Unknown" ; then

MSG="❌ 应用{APP}({ENV})部署异常，请检查"

fi

发送通知

curl -s "https://oapi.dingtalk.com/robot/send?access_token=xxx" \

-H "Content-Type: application/json" \

-d "{\"msgtype\":\"text\",\"text\":{\"content\":\"${MSG}\"}}"

11. 自动化运维平台建设

11.1 运维门户搭建

使用Backstage或自研轻量门户：

方案1：Backstage

npx @backstage/create-app@latest

cd backstage

yarn dev

方案2：轻量Web门户（Flask + Bootstrap）

功能：作业执行、审批流、操作审计、资源看板

门户核心功能模块：

|--------|----------------|
| 模块 | 功能 |
| 作业中心 | 标准化作业模板、一键执行 |
| 审批中心 | 变更申请、审批流 |
| 备份管理 | 备份列表、一键恢复 |
| 证书管理 | 证书列表、到期提醒 |
| 操作审计 | 所有操作记录、可追溯 |
| 效率度量 | 自动化率、MTTR、变更频率 |

11.2 标准化作业模板

作业模板定义（YAML）：

jobs/restart-pod.yaml

apiVersion: ops.company.com/v1

kind: OpsJob

metadata:

name: restart-pod

description: 重启指定Pod

spec:

parameters:

• name: namespace

type: string

required: true

description: 命名空间

• name: pod_name

type: string

required: true

description: Pod名称

steps:

• name: 检查Pod状态

command: "kubectl get pod {{pod_name}} -n {{namespace}}"

• name: 删除Pod触发重建

command: "kubectl delete pod {{pod_name}} -n {{namespace}}"

timeout: 60

• name: 等待Pod就绪

command: "kubectl wait pod {{pod_name}} -n {{namespace}} --for=condition=Ready --timeout=120s"

approval:

required: false

audit:

enabled: true

11.3 审批流与权限控制

审批流配置

approval_flows:

high_risk:

description: 高风险操作（删除命名空间、集群升级）

steps:

• approver: sre-lead

timeout: 1h

• approver: architect

timeout: 2h

medium_risk:

description: 中风险操作（扩容、重启节点）

steps:

• approver: sre

timeout: 30m

low_risk:

description: 低风险操作（查询、重启Pod）

auto_approve: true

RBAC权限

roles:

viewer:

permissions: read

operator:

permissions: read, execute_low_risk

sre:

permissions: read, execute_low_risk, execute_medium_risk

admin:

permissions: all

11.4 操作审计与追溯

所有操作记录到审计日志

格式：时间 | 用户 | 操作 | 目标 | 结果 | 耗时

Kubernetes审计日志

/etc/kubernetes/manifests/kube-apiserver.yaml 添加：

--audit-log-path=/var/log/k8s-audit.log

--audit-log-maxage=30

--audit-log-maxbackup=10

--audit-log-maxsize=100

--audit-policy-file=/etc/kubernetes/audit-policy.yaml

审计策略：

apiVersion: audit.k8s.io/v1

kind: Policy

rules:

• level: RequestResponse

resources:

• group: ""

resources: "pods/exec", "pods/portforward"

• level: Metadata

resources:

• group: ""

resources: "pods", "deployments", "services"

verbs: "delete", "patch", "update", "create"

• level: None

resources:

• group: ""

resources: "events"

12. 自动化验证与度量

12.1 自动化测试流水线

.gitlab-ci.yml

stages:

• lint

• test

• deploy

• verify

lint:

stage: lint

script:

• kubeval --strict k8s/*.yaml

• kube-linter lint k8s/

• helm lint charts/

test:

stage: test

script:

部署到测试环境

• helm upgrade --install test-app charts/app -f values-test.yaml -n test

等待就绪

• kubectl wait deploy/test-app -n test --for=condition=Available --timeout=120s

运行集成测试

• ./run-integration-tests.sh

性能测试

• k6 run load-test.js

verify:

stage: verify

script:

安全扫描

• trivy config --exit-code 1 --severity HIGH,CRITICAL k8s/

• kube-bench --json

合规检查

• kyverno apply policies/ -r k8s/

12.2 运维效率度量指标

|---------|------------|---------|
| 指标 | 定义 | 目标值 |
| MTTR | 平均故障恢复时间 | <30分钟 |
| 变更失败率 | 失败变更数/总变更数 | <5% |
| 部署频率 | 每周部署次数 | >10次 |
| 变更前置时间 | 代码提交到上线时间 | <1天 |
| 告警响应时间 | 告警触发到确认时间 | <5分钟 |
| 备份成功率 | 成功备份数/总备份数 | 100% |
| 恢复演练通过率 | 通过演练数/总演练数 | 100% |

度量采集脚本：

#!/bin/bash

metrics-collector.sh

echo "=== 运维效率度量 $(date) ==="

自动化率

TOTAL_OPS=$(wc -l < /var/log/ops-audit.log)

AUTO_OPS=$(grep -c "automated" /var/log/ops-audit.log)

echo "自动化率: (echo "scale=2; AUTO_OPS*100/$TOTAL_OPS" | bc)%"

MTTR（近30天）

echo "MTTR: (grep "resolved" /var/log/incidents.log --since="30 days ago" \| awk '{sum+=NF} END {print sum/NR "分钟"}')"

变更失败率

echo "变更失败率: ..."

备份成功率

echo "备份成功率: (velero backup get \| grep -c Completed)/(velero backup get | wc -l)"

12.3 自动化覆盖率统计

#!/bin/bash

coverage-report.sh

echo "=== 自动化覆盖率报告 ==="

部署自动化

echo "部署 ArgoCD应用数: $(kubectl get applications -n argocd --no-headers | wc -l)"

echo "部署 手动部署数: $(grep -c "manual deploy" /var/log/ops-audit.log)"

扩缩容自动化

echo "扩缩容 HPA数: $(kubectl get hpa -A --no-headers | wc -l)"

echo "扩缩容 KEDA数: $(kubectl get scaledobject -A --no-headers 2>/dev/null | wc -l)"

备份自动化

echo "备份 定时备份数: $(kubectl get schedules -n velero --no-headers | wc -l)"

证书自动化

echo "证书 自动管理证书数: $(kubectl get certificates -A --no-headers | wc -l)"

安全自动化

echo "安全 漏洞报告数: $(kubectl get vulnerabilityreports -A --no-headers | wc -l)"

自愈自动化

echo "自愈 自动修复次数: $(grep -c "auto-repair" /var/log/ops-audit.log)"

12.4 持续改进机制

月度复盘会议议程：

1. 度量指标回顾（自动化率、MTTR、变更失败率）

2. 本月告警TOP10分析

3. 手动操作TOP10（识别自动化机会）

4. 故障复盘（P1/P0事件）

5. 下月自动化改进计划

6. 工具链优化讨论

季度评估：

1. 自动化成熟度等级评估（L1-L5）

2. 工具链技术债清理

3. 新工具/新技术调研

4. 团队技能提升计划