K8s集群运行时异常趋势分析预警实操

K8s集群运行时异常趋势分析预警实操

|--------|------------------------------------------------------------------------------------|
| 项目 | 内容 |
| 适用版本 | Kubernetes v1.28.15 + Prometheus 2.48 + Grafana 10.2 + Loki 2.9 + containerd 1.7.x |
| 核心目标 | 运行时异常实时感知、趋势预测、容量预警、自动化应急响应 |
| 编写日期 | 2026-08-18 |
| 文档版本 | v1.0 |

1. 操作环境与前置准备

1.1 硬件要求

|--------|---------|--------|-----------|-------------------------|
| 角色 | CPU | 内存 | 硬盘 | 说明 |
| Worker | 8核+ | 16GB+ | 200GB SSD | 业务节点 |
| 监控节点 | 8核+ | 16GB+ | 500GB SSD | Prometheus+Grafana+Loki |
| 告警节点 | 4核+ | 8GB+ | 100GB SSD | Alertmanager（可复用监控节点） |

监控节点建议独立部署，避免监控组件与业务争抢资源。大规模集群（50+节点）需独立部署。

1.2 软件版本矩阵

|-----------------------|----------|-----------|
| 组件 | 版本 | 说明 |
| Kubernetes | v1.28.15 | 集群版本 |
| containerd | 1.7.11+ | 容器运行时 |
| Prometheus | 2.48.0+ | 指标存储与告警 |
| Alertmanager | 0.26.0+ | 告警路由通知 |
| Grafana | 10.2.0+ | 可视化看板 |
| Loki | 2.9.0+ | 日志聚合分析 |
| Promtail | 2.9.0+ | 日志采集器 |
| node-exporter | 1.7.0+ | 节点指标采集 |
| kube-state-metrics | 2.10.0+ | K8s资源状态指标 |
| kube-prometheus-stack | 最新 | 一键部署（可选） |

1.3 节点规划

|--------------|---------------|--------|--------------------------------------|
| 主机名 | IP地址 | 角色 | 部署组件 |
| k8s-worker01 | 192.168.10.21 | Worker | 业务Pod + node-exporter |
| k8s-worker02 | 192.168.10.22 | Worker | 业务Pod + node-exporter |
| k8s-worker03 | 192.168.10.23 | Worker | 业务Pod + node-exporter |
| k8s-monitor | 192.168.10.40 | 监控 | Prometheus/Grafana/Loki/Alertmanager |

1.4 存储规划

|--------------|----------|----------|--------------------|
| 数据类型 | 保留周期 | 预估存储 | 存储路径 |
| Loki日志 | 7天 | 200GB+ | /data/loki |
| Alertmanager | 永久 | <1GB | /data/alertmanager |
| Grafana | 永久 | <10GB | /data/grafana |

2. 运行时异常监控体系架构

2.1 异常分类与监控维度

|----------|--------------|--------------------------------|----------|
| 异常类别 | 监控对象 | 关键指标 | 预警级别 |
| 资源耗尽 | CPU/内存/磁盘/网络 | 使用率、趋势斜率、预测值 | P1/P2 |
| Pod异常 | 容器状态 | CrashLoopBackOff、OOMKilled、重启率 | P1/P2 |
| 网络异常 | 连接/丢包/延迟 | conntrack使用率、丢包率、TCP重传 | P1/P2 |
| 存储异常 | 磁盘/IO | 磁盘使用率、IO等待、inode | P1/P2 |
| 调度异常 | 调度器 | 调度失败数、Pending Pod数 | P2 |
| 应用异常 | 业务容器 | 错误率、延迟、5xx比例 | P1/P2 |

2.2 整体架构

┌──────────────────────────────────────────────────────────────────┐
│ K8s Cluster │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Worker1 │ │ Worker2 │ │ Worker3 │ │ Master │ │
│ │node-exp │ │node-exp │ │node-exp │ │kubelet │ │
│ │promtail │ │promtail │ │promtail │ │metrics │ │
│ │cAdvisor │ │cAdvisor │ │cAdvisor │ │ │ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
│ │ │ │ │ │
│ └──────────────┼──────────────┼──────────────┘ │
│ │ │ │
│ metrics │ logs │ │
│ ▼ ▼ │
│ ┌──────────────────────────┐ ┌──────────────────┐ │
│ │ Prometheus │ │ Loki │ │
│ │ (指标存储+告警规则) │ │ (日志聚合分析) │ │
│ └──────────┬───────────────┘ └────────┬─────────┘ │
│ │ alerts │ logs │
│ ▼ ▼ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Alertmanager │ │ Grafana │ │
│ │ (路由/通知/抑制) │◄──────►│ (可视化看板) │ │
│ └────────┬─────────┘ └──────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ 钉钉/企微/邮件 │ │
│ └──────────────────┘ │
└──────────────────────────────────────────────────────────────────┘

2.3 数据流转链路

采集层 存储层 分析层 通知层
───────── ───────── ───────── ─────────
node-exporter → Prometheus → 告警规则引擎 → Alertmanager → 钉钉
kubelet → Prometheus → 趋势预测模型 → Alertmanager → 企微
cAdvisor → Prometheus → 异常检测算法 → Grafana看板 → 邮件
Promtail → Loki → 日志关键字匹配 → 告警联动 → 短信
kube-state → Prometheus → 资源状态分析 → 容量预警

3. 指标监控采集部署

3.1 Prometheus部署

创建命名空间和配置：

kubectl create namespace monitoring

创建 prometheus-config.yaml：

apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-config
namespace: monitoring
data:
prometheus.yml: |
global:
scrape_interval: 15s
evaluation_interval: 15s
external_labels:
cluster: k8s-prod

alerting:
alertmanagers:

• static_configs:
• targets:
• alertmanager.monitoring.svc.cluster.local:9093

rule_files:

• /etc/prometheus/rules/*.yml

scrape_configs:

Prometheus自身

• job_name: 'prometheus'
  static_configs:
• targets: 'localhost:9090'

node-exporter

• job_name: 'node-exporter'
  kubernetes_sd_configs:
• role: endpoints
  relabel_configs:
• source_labels: __meta_kubernetes_service_name
  regex: node-exporter
  action: keep

kubelet

• job_name: 'kubelet'
  kubernetes_sd_configs:
• role: node
  scheme: https
  tls_config:
  ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
• action: labelmap
  regex: _meta_kubernetes_node_label(.+)

cAdvisor (通过kubelet)

• job_name: 'cadvisor'
  kubernetes_sd_configs:
• role: node
  scheme: https
  metrics_path: /metrics/cadvisor
  tls_config:
  ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

kube-state-metrics

• job_name: 'kube-state-metrics'
  static_configs:
• targets: 'kube-state-metrics.monitoring.svc.cluster.local:8080'

containerd (通过节点exporter或cRI)

• job_name: 'containerd'
  static_configs:
• targets: '192.168.10.21:9091','192.168.10.22:9091','192.168.10.23:9091'

部署Prometheus：

apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: prometheus
template:
metadata:
labels:
app: prometheus
spec:
serviceAccountName: prometheus
containers:

• name: prometheus
  image: harbor.local/monitoring/prometheus:v2.48.0
  args:
• --config.file=/etc/prometheus/prometheus.yml
• --storage.tsdb.path=/prometheus
• --storage.tsdb.retention.time=15d
• --storage.tsdb.retention.size=100GB
• --web.enable-lifecycle
  ports:
• containerPort: 9090
  volumeMounts:
• name: config
  mountPath: /etc/prometheus
• name: rules
  mountPath: /etc/prometheus/rules
• name: storage
  mountPath: /prometheus
  resources:
  requests:
  cpu: "2"
  memory: 4Gi
  limits:
  cpu: "4"
  memory: 8Gi
  volumes:
• name: config
  configMap:
  name: prometheus-config
• name: rules
  configMap:
  name: prometheus-rules
• name: storage
  persistentVolumeClaim:
  claimName: prometheus-pvc

---

apiVersion: v1
kind: Service
metadata:
name: prometheus
namespace: monitoring
spec:
selector:
app: prometheus
ports:

• port: 9090
  targetPort: 9090
  type: NodePort

3.2 node-exporter部署

apiVersion: apps/v1
kind: DaemonSet
metadata:
name: node-exporter
namespace: monitoring
spec:
selector:
matchLabels:
app: node-exporter
template:
metadata:
labels:
app: node-exporter
spec:
hostNetwork: true
hostPID: true
containers:

• name: node-exporter
  image: harbor.local/monitoring/node-exporter:v1.7.0
  args:
• --path.procfs=/host/proc
• --path.sysfs=/host/sys
• --path.rootfs=/host/root
• --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+)($|/)
• --collector.systemd
• --collector.processes
• --collector.tcpstat
  ports:
• containerPort: 9100
  hostPort: 9100
  volumeMounts:
• name: proc
  mountPath: /host/proc
  readOnly: true
• name: sys
  mountPath: /host/sys
  readOnly: true
• name: root
  mountPath: /host/root
  readOnly: true
  volumes:
• name: proc
  hostPath:
  path: /proc
• name: sys
  hostPath:
  path: /sys
• name: root
  hostPath:
  path: /

---

apiVersion: v1
kind: Service
metadata:
name: node-exporter
namespace: monitoring
spec:
selector:
app: node-exporter
ports:

• port: 9100
  targetPort: 9100

3.3 cAdvisor与kubelet指标

cAdvisor已集成在kubelet中，通过 /metrics/cadvisor 端点采集。关键指标：

|----------------------------------------|----------|
| 指标名 | 说明 |
| container_memory_working_set_bytes | 容器内存使用 |
| container_network_receive_bytes_total | 容器网络接收 |
| container_network_transmit_bytes_total | 容器网络发送 |
| container_fs_usage_bytes | 容器磁盘使用 |
| container_last_seen | 容器最后活跃时间 |

kubelet关键指标：

|-----------------------------------------|-------------------|
| 指标名 | 说明 |
| kubelet_running_containers | 运行中容器数 |
| kubelet_pod_start_duration_seconds | Pod启动耗时 |
| kubelet_cgroup_manager_duration_seconds | cgroup操作耗时 |
| kubelet_pleg_relist_duration_seconds | PLEG耗时（过高表示运行时异常） |

3.4 containerd指标采集

containerd内置Prometheus指标支持，需开启：

修改containerd配置，开启metrics

/etc/containerd/config.toml

metrics

address = "0.0.0.0:9091"
grpc_histogram = true

重启containerd

systemctl restart containerd

验证指标

curl http://localhost:9091/v1/metrics | head -20

containerd关键指标：

|------------------------------------------|----------|
| 指标名 | 说明 |
| containerd_grpc_request_duration_seconds | gRPC请求耗时 |
| containerd_snapshots_total | 快照数量 |
| containerd_content_total | 内容存储数量 |
| containerd_images_total | 镜像数量 |

3.5 kube-state-metrics部署

apiVersion: apps/v1
kind: Deployment
metadata:
name: kube-state-metrics
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: kube-state-metrics
template:
metadata:
labels:
app: kube-state-metrics
spec:
serviceAccountName: kube-state-metrics
containers:

• name: kube-state-metrics
  image: harbor.local/monitoring/kube-state-metrics:v2.10.0
  ports:
• containerPort: 8080
  resources:
  requests:
  cpu: 100m
  memory: 256Mi
  limits:
  cpu: 500m
  memory: 512Mi

---

apiVersion: v1
kind: Service
metadata:
name: kube-state-metrics
namespace: monitoring
spec:
selector:
app: kube-state-metrics
ports:

• port: 8080
  targetPort: 8080

4. 日志采集与异常分析

4.1 Loki部署

apiVersion: apps/v1
kind: Deployment
metadata:
name: loki
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: loki
template:
metadata:
labels:
app: loki
spec:
containers:

• name: loki
  image: harbor.local/monitoring/loki:2.9.0
  args:
• -config.file=/etc/loki/loki-config.yml
  ports:
• containerPort: 3100
  volumeMounts:
• name: config
  mountPath: /etc/loki
• name: storage
  mountPath: /data/loki
  resources:
  requests:
  cpu: "1"
  memory: 2Gi
  limits:
  cpu: "2"
  memory: 4Gi
  volumes:
• name: config
  configMap:
  name: loki-config
• name: storage
  persistentVolumeClaim:
  claimName: loki-pvc

---

apiVersion: v1
kind: ConfigMap
metadata:
name: loki-config
namespace: monitoring
data:
loki-config.yml: |
auth_enabled: false
server:
http_listen_port: 3100
common:
path_prefix: /data/loki
storage:
filesystem:
chunks_directory: /data/loki/chunks
rules_directory: /data/loki/rules
replication_factor: 1
ring:
instance_addr: 127.0.0.1
kvstore:
store: inmemory
query_range:
results_cache:
cache:
enable_fifocache: true
fifocache:
max_size_items: 1024
validity: 24h
limits_config:
retention_period: 168h
max_query_length: 721h
max_streams_per_user: 0
reject_old_samples: true
reject_old_samples_max_age: 168h
schema_config:
configs:

• from: 2024-01-01
  store: tsdb
  object_store: filesystem
  schema: v13
  index:
  prefix: index_
  period: 24h

4.2 Promtail日志采集

apiVersion: apps/v1
kind: DaemonSet
metadata:
name: promtail
namespace: monitoring
spec:
selector:
matchLabels:
app: promtail
template:
metadata:
labels:
app: promtail
spec:
serviceAccountName: promtail
containers:

• name: promtail
  image: harbor.local/monitoring/promtail:2.9.0
  args:
• -config.file=/etc/promtail/promtail-config.yml
  volumeMounts:
• name: config
  mountPath: /etc/promtail
• name: logs
  mountPath: /var/log
  readOnly: true
• name: containers
  mountPath: /var/log/containers
  readOnly: true
• name: pods
  mountPath: /var/log/pods
  readOnly: true
• name: journal
  mountPath: /var/log/journal
  readOnly: true
  volumes:
• name: config
  configMap:
  name: promtail-config
• name: logs
  hostPath:
  path: /var/log
• name: containers
  hostPath:
  path: /var/log/containers
• name: pods
  hostPath:
  path: /var/log/pods
• name: journal
  hostPath:
  path: /var/log/journal

---

apiVersion: v1
kind: ConfigMap
metadata:
name: promtail-config
namespace: monitoring
data:
promtail-config.yml: |
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:

• url: http://loki.monitoring.svc.cluster.local:3100/loki/api/v1/push
  scrape_configs:

容器日志

• job_name: kubernetes-pods
  pipeline_stages:
• docker: {}
  kubernetes_sd_configs:
• role: pod
  relabel_configs:
• source_labels: '__meta_kubernetes_pod_node_name'
  target_label: 'node'
• source_labels: '__meta_kubernetes_namespace'
  target_label: 'namespace'
• source_labels: '__meta_kubernetes_pod_name'
  target_label: 'pod'
• source_labels: '__meta_kubernetes_container_name'
  target_label: 'container'
• replacement: '/var/log/pods/*$1/*.log'
  separator: /
  source_labels:
• __meta_kubernetes_pod_uid
• __meta_kubernetes_pod_container_name
  target_label: path

containerd系统日志

• job_name: containerd-system
  static_configs:
• targets: localhost
  labels:
  job: containerd
  path: /var/log/messages
  pipeline_stages:
• match:
  selector: '{job="containerd"} |= "containerd"'
  stages:
• labels:
  runtime: containerd

kubelet日志

• job_name: kubelet-journal
  journal:
  json: false
  max_age: 7d
  path: /var/log/journal
  labels:
  job: kubelet
  relabel_configs:
• source_labels: '__journal__systemd_unit'
  target_label: 'unit'

4.3 运行时日志关键字监控

在Loki中配置告警规则，监控运行时异常关键字：

/etc/loki/rules/fake/runtime-alerts.yml

groups:

• name: runtime-errors
  rules:

• alert: ContainerdCriError
  expr: |
  count_over_time({job="containerd"} |= "error" |= "cri" 5m) > 5
  for: 2m
  labels:
  severity: critical
  annotations:
  summary: "containerd CRI错误激增"
  description: "节点{{ $labels.node }} containerd CRI错误5分钟内超过5次"

• alert: KubeletPlegHigh
  expr: |
  count_over_time({job="kubelet", unit="kubelet.service"} |= "PLEG" |= "unhealthy" 5m) > 0
  for: 1m
  labels:
  severity: critical
  annotations:
  summary: "kubelet PLEG不健康"
  description: "节点{{ $labels.node }} kubelet PLEG异常，可能导致Pod无法调度"

• alert: OomKilledDetected
  expr: |
  count_over_time({namespace=~".+"} |= "OOMKilled" 10m) > 0
  for: 0m
  labels:
  severity: warning
  annotations:
  summary: "检测到OOMKilled"
  description: "Pod {{ $labels.pod }} 因OOM被杀死"

• alert: CrashLoopBackOff
  expr: |
  count_over_time({namespace=~".+"} |= "CrashLoopBackOff" 5m) > 0
  for: 1m
  labels:
  severity: warning
  annotations:
  summary: "Pod CrashLoopBackOff"
  description: "Pod {{ $labels.pod }} 进入CrashLoopBackOff状态"

4.4 日志异常模式匹配

常用异常日志模式：

|-------------|---------------------------------------------------|--------|
| 异常模式 | 关键字 | 级别 |
| CRI调用失败 | failed to get sandbox / rpc error | P1 |
| kubelet节点异常 | PLEG is not healthy / runtime operation timed out | P0 |
| 镜像拉取失败 | Failed to pull image / manifest unknown | P2 |
| OOM内存溢出 | OOMKilled / out of memory | P1 |
| 磁盘压力 | disk pressure / image garbage collection | P1 |
| 网络异常 | network plugin cni / failed to setup network | P1 |
| 调度失败 | FailedScheduling / insufficient cpu/memory | P2 |

Loki查询示例：

查询所有运行时错误

{job=~"containerd|kubelet"} |= "error" != "EOF"

按节点统计错误数

sum by (node) (count_over_time({job="containerd"} |= "error" 1h))

趋势对比（同比上周）

sum(count_over_time({namespace="prod"} |= "500" 1h))
/
sum(count_over_time({namespace="prod"} |= "500" 1h offset 168h))

5. 运行时健康监控体系

5.1 containerd健康指标

查看containerd服务状态

systemctl is-active containerd
systemctl show containerd --property=NRestarts,ActiveEnterTimestamp

查看containerd指标

curl -s http://localhost:9091/v1/metrics | grep -E 'grpc_requests|grpc_request_duration'

Prometheus查询

containerd gRPC错误率

rate(containerd_grpc_requests_total{code=~"Internal|Unavailable|Unknown"}5m)

containerd gRPC延迟P99

histogram_quantile(0.99, rate(containerd_grpc_request_duration_seconds_bucket5m))

5.2 kubelet健康监控

kubelet健康检查

curl -sk https://localhost:10250/healthz

应返回 ok

kubelet就绪检查

curl -sk https://localhost:10250/readyz

PLEG耗时（超过3s异常）

curl -sk https://localhost:10250/metrics | grep kubelet_pleg_relist_duration

Prometheus告警表达式

PLEG P99延迟 > 3s

histogram_quantile(0.99, rate(kubelet_pleg_relist_duration_seconds_bucket5m)) > 3

kubelet重启

changes(kubelet_running_pods5m) < 0 and time() - kubelet_node_start_timestamp < 300

5.3 节点资源趋势监控

关键Prometheus查询：

CPU使用率（按节点）

100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}5m)) * 100)

内存使用率

100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes))

磁盘使用率

100 - (100 * node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"})

磁盘IO等待

rate(node_pressure_io_waiting_seconds_total5m)

网络接收速率

rate(node_network_receive_bytes_total{device!~"lo|cali.*|tunl0|vxlan"}5m) * 8

网络发送速率

rate(node_network_transmit_bytes_total{device!~"lo|cali.*|tunl0|vxlan"}5m) * 8

TCP重传率

rate(node_netstat_Tcp_RetransSegs5m) / rate(node_netstat_Tcp_OutSegs5m) * 100

conntrack使用率

node_nf_conntrack_entries / node_nf_conntrack_entries_limit * 100

5.4 Pod异常状态监控

CrashLoopBackOff Pod数

sum(kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}) by (namespace, pod)

OOMKilled次数

increase(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}1h)

Pending Pod数（调度失败）

sum(kube_pod_status_phase{phase="Pending"}) by (namespace)

容器重启率（5分钟内重启>2次）

increase(kube_pod_container_status_restarts_total5m) > 2

Pod不可用

sum(kube_pod_status_phase{phase=~"Failed|Unknown"}) by (namespace, pod)

容器CPU throttling（被限流）

rate(container_cpu_cfs_throttled_seconds_total5m) > 0

5.5 网络异常监控

conntrack使用率 > 80%

node_nf_conntrack_entries / node_nf_conntrack_entries_limit > 0.8

TCP重传率 > 5%

rate(node_netstat_Tcp_RetransSegs5m) / rate(node_netstat_Tcp_OutSegs5m) > 0.05

网卡丢包率

rate(node_network_receive_drop_total5m) > 10
rate(node_network_transmit_drop_total5m) > 10

连接数异常增长

increase(node_netstat_Tcp_CurrEstab10m) > 1000

Calico节点异常

calico Felix健康检查失败数

6. 异常趋势分析模型

6.1 静态阈值告警

最基础的告警方式，适用于明确的资源上限：

• alert: NodeCpuHigh
  expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}5m)) * 100) > 85
  for: 10m
  labels:
  severity: warning
  annotations:
  summary: "节点CPU使用率超过85%"

• alert: NodeMemoryCritical
  expr: 100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 95
  for: 5m
  labels:
  severity: critical

6.2 动态基线告警

基于历史同期数据计算动态阈值，适用于业务有明显周期性的场景：

当前值 vs 上周同期基线（偏差>50%告警）

(
avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}5m))

avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}5m offset 168h))
)
/
avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}5m offset 168h))
> 0.5

6.3 同比环比趋势分析

环比（当前小时 vs 上一小时）

sum(rate(container_cpu_usage_seconds_total1h))
/
sum(rate(container_cpu_usage_seconds_total1h offset 1h))

同比（当前 vs 上周同时段）

sum(rate(container_network_receive_bytes_total1h))
/
sum(rate(container_network_receive_bytes_total1h offset 168h))

7日移动平均

avg_over_time((sum by(instance) (rate(node_cpu_seconds_total{mode!="idle"}5m)))7d:)

6.4 异常突变检测

使用3-sigma原则检测异常突变：

当前值偏离7日均值超过3倍标准差

(
sum by(instance) (rate(node_cpu_seconds_total{mode!="idle"}5m))

avg_over_time((sum by(instance) (rate(node_cpu_seconds_total{mode!="idle"}5m)))7d:)
)
>
3 * stddev_over_time((sum by(instance) (rate(node_cpu_seconds_total{mode!="idle"}5m)))7d:)

6.5 容量趋势预测

线性回归预测资源耗尽时间：

磁盘使用率趋势预测（预测24小时后使用率）

predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}6h, 24*3600) < 0

内存耗尽预测（预测何时内存不足）

predict_linear(node_memory_MemAvailable_bytes6h, 3600) < 0

conntrack表满预测

predict_linear(node_nf_conntrack_entries1h, 3600) > node_nf_conntrack_entries_limit * 0.95

按当前增长率预测磁盘满的时间（小时）

(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"})
/
(-1 * deriv(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}1h))
/ 3600 < 24

7. 告警规则配置

7.1 运行时异常告警规则

创建 prometheus-rules.yaml：

apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-rules
namespace: monitoring
data:
runtime-alerts.yml: |
groups:

• name: runtime-health
  rules:

containerd服务异常

• alert: ContainerdDown
  expr: up{job="containerd"} == 0
  for: 1m
  labels:
  severity: critical
  annotations:
  summary: "containerd服务停止"
  description: "节点{{ $labels.instance }} containerd已停止超过1分钟"

containerd gRPC错误率

• alert: ContainerdGrpcErrorHigh
  expr: |
  rate(containerd_grpc_requests_total{code=~"Internal|Unavailable|Unknown|DeadlineExceeded"}5m)
  /
  rate(containerd_grpc_requests_total5m) > 0.05
  for: 5m
  labels:
  severity: warning
  annotations:
  summary: "containerd gRPC错误率过高"
  description: "节点{{ $labels.instance }} gRPC错误率超过5%"

containerd gRPC延迟

• alert: ContainerdGrpcLatencyHigh
  expr: histogram_quantile(0.99, rate(containerd_grpc_request_duration_seconds_bucket5m)) > 2
  for: 5m
  labels:
  severity: warning
  annotations:
  summary: "containerd gRPC延迟过高"
  description: "节点{{ $labels.instance }} P99延迟超过2秒"

kubelet异常

• alert: KubeletDown
  expr: up{job="kubelet"} == 0
  for: 1m
  labels:
  severity: critical
  annotations:
  summary: "kubelet服务停止"

PLEG异常

• alert: KubeletPlegHighLatency
  expr: histogram_quantile(0.99, rate(kubelet_pleg_relist_duration_seconds_bucket5m)) > 3
  for: 5m
  labels:
  severity: critical
  annotations:
  summary: "kubelet PLEG延迟过高"
  description: "节点{{ $labels.instance }} PLEG P99延迟超过3秒，Pod可能无法正常调度"

kubelet重启

• alert: KubeletRestarted
  expr: changes(kubelet_node_start_timestamp15m) > 0
  for: 0m
  labels:
  severity: warning
  annotations:
  summary: "kubelet发生重启"

7.2 资源趋势告警规则

resource-alerts.yml: |
groups:

• name: resource-trend
  rules:

CPU高

• alert: NodeCpuUsageHigh
  expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}5m)) * 100) > 85
  for: 10m
  labels:
  severity: warning
  annotations:
  summary: "节点CPU使用率超过85%"
  description: "节点{{ labels.instance }} CPU使用率{{ value }}%"

CPU持续高（趋势）

• alert: NodeCpuSustainedHigh
  expr: avg_over_time((100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}5m)) * 100))30m:) > 80
  for: 30m
  labels:
  severity: critical
  annotations:
  summary: "节点CPU持续高负载"

内存高

• alert: NodeMemoryHigh
  expr: 100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 90
  for: 10m
  labels:
  severity: warning

磁盘即将满（预测）

• alert: DiskWillFillIn24h
  expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay",mountpoint="/"}6h, 24*3600) < 0
  for: 1h
  labels:
  severity: critical
  annotations:
  summary: "磁盘预计24小时内写满"
  description: "节点{{ $labels.instance }} 根分区预计24小时内耗尽"

磁盘使用率高

• alert: NodeDiskHigh
  expr: 100 - (100 * node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}) > 85
  for: 10m
  labels:
  severity: warning

内存耗尽预测

• alert: MemoryWillExhaust
  expr: predict_linear(node_memory_MemAvailable_bytes6h, 3600) < 0
  for: 15m
  labels:
  severity: critical
  annotations:
  summary: "内存预计1小时内耗尽"

7.3 网络异常告警规则

network-alerts.yml: |
groups:

• name: network-anomaly
  rules:

conntrack使用率

• alert: ConntrackHigh
  expr: node_nf_conntrack_entries / node_nf_conntrack_entries_limit > 0.8
  for: 5m
  labels:
  severity: warning
  annotations:
  summary: "conntrack使用率超过80%"

conntrack即将满（预测）

• alert: ConntrackWillFull
  expr: predict_linear(node_nf_conntrack_entries1h, 1800) > node_nf_conntrack_entries_limit * 0.95
  for: 10m
  labels:
  severity: critical

TCP重传率

• alert: TcpRetransHigh
  expr: rate(node_netstat_Tcp_RetransSegs5m) / rate(node_netstat_Tcp_OutSegs5m) > 0.05
  for: 10m
  labels:
  severity: warning
  annotations:
  summary: "TCP重传率超过5%"

网卡丢包

• alert: NetworkPacketLoss
  expr: rate(node_network_receive_drop_total5m) > 10 or rate(node_network_transmit_drop_total5m) > 10
  for: 5m
  labels:
  severity: warning

网络带宽跑满

• alert: NetworkBandwidthFull
  expr: (rate(node_network_receive_bytes_total{device!~"lo|cali.*|tunl0|vxlan"}5m) * 8) > (node_network_speed_bytes{device!~"lo|cali.*|tunl0|vxlan"} * 0.9 * 8)
  for: 10m
  labels:
  severity: warning

7.4 告警分级与抑制

|------------|-----------|----------|----------|
| 级别 | 触发条件 | 响应时间 | 通知方式 |
| P1 Warning | 性能下降、资源预警 | 30分钟 | 钉钉+邮件 |
| P2 Info | 趋势异常、容量预测 | 2小时 | 邮件+看板 |

告警抑制规则（Alertmanager）：

inhibit_rules:

P0告警抑制同节点P1/P2告警

• source_match:
  severity: critical
  target_match:
  severity: warning
  equal: 'instance', 'alertname'

节点宕机抑制该节点所有Pod告警

• source_match:
  alertname: NodeDown
  target_match_re:
  alertname: 'Pod.*'
  equal: 'instance'

8. Alertmanager告警通知

8.1 Alertmanager部署

apiVersion: apps/v1
kind: Deployment
metadata:
name: alertmanager
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: alertmanager
template:
metadata:
labels:
app: alertmanager
spec:
containers:

• name: alertmanager
  image: harbor.local/monitoring/alertmanager:v0.26.0
  args:
• --config.file=/etc/alertmanager/alertmanager.yml
• --storage.path=/alertmanager
• --web.external-url=http://alertmanager:9093
  ports:
• containerPort: 9093
  volumeMounts:
• name: config
  mountPath: /etc/alertmanager
• name: storage
  mountPath: /alertmanager
  volumes:
• name: config
  configMap:
  name: alertmanager-config
• name: storage
  emptyDir: {}

---

apiVersion: v1
kind: Service
metadata:
name: alertmanager
namespace: monitoring
spec:
selector:
app: alertmanager
ports:

• port: 9093
  targetPort: 9093

8.2 告警路由配置

apiVersion: v1
kind: ConfigMap
metadata:
name: alertmanager-config
namespace: monitoring
data:
alertmanager.yml: |
global:
resolve_timeout: 5m

企业微信webhook

wechat_api_url: 'https://qyapi.weixin.qq.com/cgi-bin/'
wechat_api_secret: '<企业微信Secret>'
wechat_api_corp_id: '<企业ID>'

route:
group_by: 'alertname', 'cluster', 'instance'
group_wait: 10s
group_interval: 5m
repeat_interval: 4h
receiver: 'default'
routes:

P0告警立即通知

• match:
  severity: critical
  receiver: 'critical-webhook'
  group_wait: 0s
  repeat_interval: 1h

P1告警

• match:
  severity: warning
  receiver: 'warning-dingtalk'
  repeat_interval: 4h

P2告警

• match:
  severity: info
  receiver: 'info-email'
  repeat_interval: 12h

receivers:

• name: 'default'
  webhook_configs:

• url: 'http://alertmanager-webhook.monitoring.svc.cluster.local:8080/webhook'

• name: 'critical-webhook'
  webhook_configs:

• url: 'http://alertmanager-webhook.monitoring.svc.cluster.local:8080/critical'
  send_resolved: true

• name: 'warning-dingtalk'
  webhook_configs:

• url: 'http://alertmanager-webhook.monitoring.svc.cluster.local:8080/dingtalk'
  send_resolved: true

• name: 'info-email'
  email_configs:

• to: 'ops@company.com'
  from: 'alert@company.com'
  smarthost: 'smtp.company.com:587'
  auth_username: 'alert@company.com'
  auth_password: '<邮箱密码>'
  send_resolved: true

inhibit_rules:

• source_match:
  severity: critical
  target_match:
  severity: warning
  equal: 'instance'

8.3 钉钉/企业微信通知

部署Webhook转发服务（支持钉钉和企业微信）：

#!/usr/bin/env python3

alertmanager-webhook.py

from flask import Flask, request
import requests
import json

app = Flask(name)

DINGTALK_WEBHOOK = "https://oapi.dingtalk.com/robot/send?access_token=\<token>"
WECHAT_WEBHOOK = "https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=\<key>"

def send_dingtalk(alert):
severity = alert'labels'.get('severity', 'unknown')
emoji = "🔴" if severity == "critical" else "��" if severity == "warning" else "🔵"
content = f"""{emoji} **{alert'annotations'.get('summary', '告警')}**
> 级别: {severity}
> 节点: {alert'labels'.get('instance', 'N/A')}
> 描述: {alert'annotations'.get('description', 'N/A')}
> 时间: {alert'startsAt'}"""

data = {
"msgtype": "markdown",
"markdown": {"title": "K8s告警", "text": content}
}
requests.post(DINGTALK_WEBHOOK, json=data)

def send_wechat(alert):
severity = alert'labels'.get('severity', 'unknown')
content = f"{severity.upper()} {alert'annotations'.get('summary', '告警')}\n节点: {alert'labels'.get('instance', 'N/A')}\n描述: {alert'annotations'.get('description', 'N/A')}"
data = {"msgtype": "text", "text": {"content": content}}
requests.post(WECHAT_WEBHOOK, json=data)

@app.route('/webhook', methods='POST')
def webhook():
data = request.json
for alert in data.get('alerts', \[\]):
if alert'status' == 'firing':
send_dingtalk(alert)
send_wechat(alert)
return "ok"

@app.route('/critical', methods='POST')
def critical():
data = request.json
for alert in data.get('alerts', \[\]):
if alert'status' == 'firing':
send_dingtalk(alert)
send_wechat(alert)
return "ok"

@app.route('/dingtalk', methods='POST')
def dingtalk():
data = request.json
for alert in data.get('alerts', \[\]):
if alert'status' == 'firing':
send_dingtalk(alert)
return "ok"

if name == 'main':
app.run(host='0.0.0.0', port=8080)

8.4 告警聚合与抑制

同一告警分组聚合（避免告警风暴）

route:
group_by: 'alertname', 'cluster'
group_wait: 30s # 首次告警等待30s，聚合同组告警
group_interval: 5m # 同组新告警间隔5分钟
repeat_interval: 4h # 重复告警间隔4小时

静默规则（维护期间静默）

通过Alertmanager API创建

curl -X POST http://alertmanager:9093/api/v2/silences -d '{

"matchers": {"name":"severity","value":"warning","isRegex":false},

"startsAt": "2026-08-18T22:00:00.000Z",

"endsAt": "2026-08-19T02:00:00.000Z",

"createdBy": "ops",

"comment": "夜间维护窗口"

}'

9. Grafana可视化看板

9.1 Grafana部署与配置

apiVersion: apps/v1
kind: Deployment
metadata:
name: grafana
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: grafana
template:
metadata:
labels:
app: grafana
spec:
containers:

• name: grafana
  image: harbor.local/monitoring/grafana:10.2.0
  ports:
• containerPort: 3000
  env:
• name: GF_SECURITY_ADMIN_PASSWORD
  value: "Grafana@2026"
• name: GF_USERS_ALLOW_SIGN_UP
  value: "false"
  volumeMounts:
• name: storage
  mountPath: /var/lib/grafana
• name: datasources
  mountPath: /etc/grafana/provisioning/datasources
  resources:
  requests:
  cpu: 500m
  memory: 1Gi
  limits:
  cpu: "2"
  memory: 2Gi
  volumes:
• name: storage
  persistentVolumeClaim:
  claimName: grafana-pvc
• name: datasources
  configMap:
  name: grafana-datasources

---

apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-datasources
namespace: monitoring
data:
datasources.yml: |
apiVersion: 1
datasources:

• name: Prometheus
  type: prometheus
  access: proxy
  url: http://prometheus.monitoring.svc.cluster.local:9090
  isDefault: true
• name: Loki
  type: loki
  access: proxy
  url: http://loki.monitoring.svc.cluster.local:3100

9.2 运行时总览看板

关键面板配置：

|--------------|-------------------------------------------------------------------------------------------------------------------|----------|
| 面板 | 查询语句 | 图表类型 |
| containerd状态 | up{job="containerd"} | Stat |
| kubelet状态 | up{job="kubelet"} | Stat |
| 集群CPU使用率 | 100 - (avg(rate(node_cpu_seconds_total{mode="idle"}5m)) * 100) | Gauge |
| 集群内存使用率 | 100 * (1 - (sum(node_memory_MemAvailable_bytes) / sum(node_memory_MemTotal_bytes))) | Gauge |
| 集群磁盘使用率 | 100 - (100 * sum(node_filesystem_avail_bytes{mountpoint="/"}) / sum(node_filesystem_size_bytes{mountpoint="/"})) | Gauge |
| Pod总数 | sum(kube_pod_status_phase{phase="Running"}) | Stat |
| 异常Pod数 | `sum(kube_pod_status_phase{phase=~"Failed | Pending |
| 运行时重启数 | increase(kube_pod_container_status_restarts_total1h) | Table |

9.3 趋势分析看板

|-------------|----------------------------------------------------------------------------|-------------|
| 面板 | 查询语句 | 图表类型 |
| 内存趋势（7天） | 100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) | Time series |
| 网络入流量趋势 | `rate(node_network_receive_bytes_total{device!~"lo | .*cali.* |
| 网络出流量趋势 | `rate(node_network_transmit_bytes_total{device!~"lo | .*cali.* |
| conntrack趋势 | node_nf_conntrack_entries / node_nf_conntrack_entries_limit * 100 | Time series |
| TCP连接数趋势 | node_netstat_Tcp_CurrEstab | Time series |
| 磁盘IO趋势 | rate(node_disk_io_time_seconds_total5m) * 100 | Time series |
| 同比对比 | 当前值 / 上周同期值 | Time series |

9.4 异常告警看板

|-----------|-----------------------|
| 面板 | 内容 |
| 告警趋势（24h） | 按级别统计告警数 |
| TOP10告警源 | 按instance/alertname统计 |
| 告警响应时长 | 从firing到resolved的时间差 |
| 告警静默列表 | 当前生效的silence |
| P0告警历史 | critical级别告警记录 |

10. 趋势预测与容量预警

10.1 资源使用趋势预测

CPU使用率线性回归预测（1小时后）

predict_linear(
(100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}5m)) * 100))6h:,
3600
)

内存使用预测（2小时后）

predict_linear(
(100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)))6h:,
7200
)

基于7天数据的趋势预测

predict_linear(
(sum by(instance) (rate(node_cpu_seconds_total{mode!="idle"}1h)))7d:,
86400
)

10.2 磁盘容量预警

节点磁盘监控脚本（配合cron定时执行）

#!/bin/bash
THRESHOLD=80
for node in (kubectl get nodes -o jsonpath='{.items\[\*\].status.addresses\[?(@.type=="InternalIP")\].address}'); do USAGE=(ssh root@{node} "df -h / \| awk 'NR==2{print \\5}' | tr -d '%'")
if ${USAGE} -gt ${THRESHOLD} ; then

计算按当前增速耗尽时间

REMAIN=(ssh root@{node} "df -B1 / | awk 'NR==2{print \4}'") GROWTH=(ssh root@{node} "cat /tmp/disk_growth_rate") HOURS_LEFT=((REMAIN / GROWTH / 3600))
echo "WARN: {node} 磁盘使用率{USAGE}%, 预计${HOURS_LEFT}小时后耗尽"

发送告警...

fi
done

Prometheus磁盘预测告警：

• alert: DiskPredictionCritical
  expr: |
  predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay",mountpoint="/"}6h, 86400) < 0
  for: 30m
  labels:
  severity: critical
  annotations:
  summary: "磁盘预计24小时内写满"
  description: "节点{{ $labels.instance }} 根分区按当前增速预计24小时内耗尽，请立即清理"

• alert: DiskPredictionWarning
  expr: |
  predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay",mountpoint="/"}24h, 259200) < 0
  for: 1h
  labels:
  severity: warning
  annotations:
  summary: "磁盘预计3天内写满"

10.3 连接数趋势预警

conntrack表满预测（30分钟内超过95%）

predict_linear(node_nf_conntrack_entries1h, 1800) > node_nf_conntrack_entries_limit * 0.95

TCP连接数异常增长（1小时内增长超过50%）

(
node_netstat_Tcp_CurrEstab

node_netstat_Tcp_CurrEstab offset 1h
)
/
node_netstat_Tcp_CurrEstab offset 1h > 0.5

容器网络连接数趋势

sum by(pod) (rate(container_network_tcp_connections_total5m))

10.4 自动化容量扩缩容建议

基于趋势数据生成扩容建议：

#!/bin/bash

容量评估脚本

echo "========== K8s集群容量评估报告 =========="
echo "生成时间: $(date '+%Y-%m-%d %H:%M:%S')"
echo ""

1. CPU容量

CPU_TOTAL=(kubectl get nodes -o jsonpath='{.items\[\*\].status.capacity.cpu}' \| tr ' ' '\\n' \| awk '{sum+=1}END{print sum}')
CPU_USED=(kubectl top nodes --no-headers \| awk '{sum+=3}END{print sum}' | tr -d '%')
echo "CPU 总核数: {CPU_TOTAL}, 已用: {CPU_USED}%"

2. 内存容量

MEM_TOTAL=(kubectl get nodes -o jsonpath='{.items\[\*\].status.capacity.memory}' \| tr ' ' '\\n' \| sed 's/Ki//' \| awk '{sum+=1}END{print sum/1024/1024 "Gi"}')
MEM_USED=(kubectl top nodes --no-headers \| awk '{sum+=5}END{print sum}' | tr -d '%')
echo "内存 总量: {MEM_TOTAL}, 已用: {MEM_USED}%"

3. Pod容量

POD_TOTAL=(kubectl get nodes -o jsonpath='{.items\[\*\].status.capacity.pods}' \| tr ' ' '\\n' \| awk '{sum+=1}END{print sum}')
POD_USED=(kubectl get pods -A --no-headers \| wc -l) echo "\[Pod\] 总容量: {POD_TOTAL}, 已用: {POD_USED} (((POD_USED*100/POD_TOTAL))%)"

4. 扩容建议

echo ""
echo "========== 扩容建议 =========="
if ${CPU_USED} -gt 70 ; then
echo "⚠ CPU使用率超过70%，建议新增Worker节点"
fi
if ${MEM_USED} -gt 70 ; then
echo "⚠ 内存使用率超过70%，建议新增Worker节点"
fi
if $((POD_USED\*100/POD_TOTAL)) -gt 75 ; then
echo "⚠ Pod容量使用率超过75%，建议新增Worker节点"
fi
echo "✅ 容量评估完成"

11. 应急响应流程

11.1 告警分级响应矩阵

|--------|---------------------------------|----------|---------|--------------|
| 级别 | 告警类型 | 响应时间 | 处理人 | 升级条件 |
| P1 | CPU/内存>90%、磁盘预测24h满、OOM、PLEG异常 | 30分钟 | 值班SRE | 1小时未解决→技术负责人 |
| P2 | CPU/内存>80%、TCP重传高、调度失败、趋势异常 | 2小时 | SRE团队 | 4小时未解决→技术负责人 |
| P3 | 容量预测、日志异常、性能基线偏离 | 工作日 | SRE团队 | - |

11.2 运行时异常应急处理

containerd停止应急：

1. 确认状态

systemctl status containerd
crictl info

2. 查看日志

journalctl -u containerd -n 200 --no-pager

3. 尝试重启

systemctl restart containerd

4. 如重启失败，检查配置

containerd config dump > /dev/null

5. 检查残留进程

ps aux | grep containerd
ss -lntp | grep containerd.sock

6. 紧急恢复（清理残留后重启）

rm -f /run/containerd/containerd.sock
systemctl restart containerd

7. 验证恢复

crictl ps
kubectl get nodes

节点NotReady应急：

1. 确认节点状态

kubectl get nodes
kubectl describe node <node-name>

2. 检查kubelet

systemctl status kubelet
journalctl -u kubelet -n 100

3. 检查containerd

systemctl status containerd
crictl info

4. 检查资源

free -h
df -h
dmesg | tail -50

5. 紧急驱逐（如节点无法恢复）

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --force

6. 恢复后重新调度

kubectl uncordon <node-name>

11.3 自动化自愈脚本

#!/bin/bash

k8s-node-auto-heal.sh

节点自动健康检查与自愈

LOGFILE="/var/log/k8s-heal.log"

log() {
echo "$(date '+%Y-%m-%d %H:%M:%S') 1" \>\> {LOGFILE}
}

1. containerd健康检查

if ! systemctl is-active containerd > /dev/null; then
log "containerd未运行，尝试重启..."
systemctl restart containerd
sleep 5
if systemctl is-active containerd > /dev/null; then
log "containerd重启成功"
else
log "containerd重启失败，需人工介入"

发送告警...

fi
fi

2. kubelet健康检查

if ! systemctl is-active kubelet > /dev/null; then
log "kubelet未运行，尝试重启..."
systemctl restart kubelet
fi

3. 磁盘空间检查

DISK_USAGE=(df / \| awk 'NR==2{print 5}' | tr -d '%')
if ${DISK_USAGE} -gt 90 ; then
log "磁盘使用率${DISK_USAGE}%，执行清理..."

清理containerd未使用镜像

crictl rmi --prune

清理journal日志

journalctl --vacuum-size=500M

清理/tmp

find /tmp -type f -atime +7 -delete
fi

4. conntrack检查

CT_COUNT=(sysctl -n net.netfilter.nf_conntrack_count) CT_MAX=(sysctl -n net.netfilter.nf_conntrack_max)
if $((CT_COUNT \* 100 / CT_MAX)) -gt 90 ; then
log "conntrack使用率过高，调大上限..."
sysctl -w net.netfilter.nf_conntrack_max=2097152
fi

5. OOM Pod检查

OOM_PODS=(crictl ps -a \| grep OOMKilled \| wc -l) if \[ {OOM_PODS} -gt 0 ]; then
log "检测到${OOM_PODS}个OOMKilled容器"
fi

配置定时执行：

每5分钟执行一次

chmod +x /opt/k8s-node-auto-heal.sh
(crontab -l 2>/dev/null; echo "*/5 * * * * /opt/k8s-node-auto-heal.sh") | crontab -

11.4 事后复盘与规则优化

复盘模板：

【告警复盘报告】
告警名称：
发生时间：
恢复时间：
持续时长：
影响范围：
根本原因：
处理过程：
改进措施：

1. 短期（立即执行）：
2. 中期（1周内）：
3. 长期（1月内）：
   告警规则优化：

• 阈值调整：
• 新增告警：
• 移除误报告警：