【Prometheus·告警篇】告警规则：Recording Rules 与 Alerting Rules 最佳实践

前言

Prometheus 的规则分为两类：Recording Rules（预计算指标）和 Alerting Rules（告警规则）。写好规则是做监控告警的核心技能。本篇详解两类规则的语法、设计原则和常见模式。

一、Recording Rules：预计算指标

为什么需要 Recording Rules

promql 复制代码

# 复杂查询
sum(rate(http_requests_total{method="GET"}[5m])) by (service, status) 
  / 
sum(rate(http_requests_total[5m])) by (service) * 100

# 问题：
# 1. 每次查询都要重新计算
# 2. Dashboard 频繁查询 → Prometheus 负载高
# 3. 范围查询更慢（要逐点计算）

promql 复制代码

# Recording Rule 预计算
# http_request_error_rate:precomputed → 存入 TSDB
# 查询时直接读预计算结果，O(1)

语法

yaml 复制代码

# rules/recording.yml
groups:
  - name: http_metrics
    interval: 30s          # 每 30 秒评估一次
    rules:
      - record: job:http_requests:rate5m       # 指标名
        expr: sum by (job)(rate(http_requests_total[5m]))
        # 可选：添加标签
        labels:
          team: platform

      - record: job:http_requests:error_rate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (job) 
          / 
          sum(rate(http_requests_total[5m])) by (job) * 100

      - record: instance:cpu_usage:ratio
        expr: |
          1 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance))

命名约定

复制代码

规则名格式：
  <scope>:<metric>:<operation>

示例：
  job:http_requests:rate5m          # 范围:指标:操作
  instance:cpu_usage:ratio           # 实例级
  cluster:memory_usage:percentage   # 集群级

使用预计算指标

promql 复制代码

# 查询预计算结果
job:http_requests:rate5m

# Dashboard 查询更快
job:http_requests:error_rate > 5

# 告警规则用预计算指标
expr: job:http_requests:error_rate > 5

完整示例

yaml 复制代码

# rules/recording_rules.yml
groups:
  # ===== HTTP 指标 =====
  - name: http_recording
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job, method)(rate(http_requests_total[5m]))

      - record: job:http_requests:rate1m
        expr: sum by (job, method)(rate(http_requests_total[1m]))

      - record: job:http_requests:error_rate5m
        expr: |
          sum by (job)(rate(http_requests_total{status=~"5.."}[5m])) 
          / sum by (job)(rate(http_requests_total[5m])) * 100

      - record: job:http_request_duration:p99
        expr: |
          histogram_quantile(0.99, 
            sum by (job, le)(rate(http_request_duration_seconds_bucket[5m])))

      - record: job:http_request_duration:p95
        expr: |
          histogram_quantile(0.95,
            sum by (job, le)(rate(http_request_duration_seconds_bucket[5m])))

  # ===== 资源指标 =====
  - name: resource_recording
    interval: 30s
    rules:
      - record: instance:cpu_usage:ratio
        expr: 1 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])))

      - record: instance:memory_usage:ratio
        expr: 1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

      - record: instance:disk_usage:ratio
        expr: 1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes)

      - record: instance:network_rx:rate5m
        expr: rate(node_network_receive_bytes_total{device!~"lo|docker.*|veth.*"}[5m]) * 8

      - record: instance:network_tx:rate5m
        expr: rate(node_network_transmit_bytes_total{device!~"lo|docker.*|veth.*"}[5m]) * 8

  # ===== Kubernetes 指标 =====
  - name: k8s_recording
    interval: 30s
    rules:
      - record: namespace:pod_count
        expr: count by(namespace)(kube_pod_status_phase{phase="Running"})

      - record: namespace:cpu_request:total
        expr: sum by(namespace)(kube_pod_container_resource_requests{resource="cpu"})

      - record: namespace:memory_usage:total
        expr: sum by(namespace)(container_memory_working_set_bytes{container!=""})

二、Alerting Rules：告警规则

语法

yaml 复制代码

# rules/alerting.yml
groups:
  - name: alerting_rules
    interval: 15s
    rules:
      - alert: HighCPUUsage           # 告警名（唯一）
        expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m                        # 持续 5 分钟才告警
        labels:
          severity: warning           # 告警标签
        annotations:
          summary: "CPU usage > 80% on {{ $labels.instance }}"
          description: "CPU usage is {{ $value | printf \"%.2f\" }}% (threshold: 80%)"
          runbook_url: "https://wiki.example.com/runbooks/high-cpu"

关键字段

字段	说明
alert	告警名称，唯一标识
expr	PromQL 表达式，返回非空则触发
for	持续时间，防止瞬时波动误告警
labels	附加标签，用于路由
annotations	描述信息，支持模板

for 的三阶段

复制代码

告警生命周期：
  Pending（等待中）→ Firing（触发中）→ Resolved（已恢复）

  expr 成立 → Pending → for 时间后仍成立 → Firing → 发送告警
  expr 不成立 → Resolved → 发送恢复通知（send_resolved=true）

示例：for: 5m

  10:00  expr 成立 → Pending
  10:03  expr 仍成立 → 继续 Pending
  10:05  仍成立 → Firing → 发送告警
  10:08  expr 不成立 → Resolved → 发送恢复通知

三、告警规则最佳实践

1. 分级告警

yaml 复制代码

groups:
  - name: severity-based
    rules:
      # Critical：立即处理
      - alert: ServiceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Service {{ $labels.job }} is down"

      # Warning：关注但不紧急
      - alert: HighCPU
        expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High CPU on {{ $labels.instance }}"

      # Info：仅记录
      - alert: DiskSpaceLow
        expr: (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 70
        for: 30m
        labels:
          severity: info
        annotations:
          summary: "Disk space low on {{ $labels.instance }}"

2. 使用预计算指标

yaml 复制代码

# ❌ 直接复杂查询（慢）
- alert: HighErrorRate
  expr: |
    sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
    / sum(rate(http_requests_total[5m])) by (service) * 100 > 5
  for: 5m

# ✅ 用 Recording Rule（快）
- alert: HighErrorRate
  expr: job:http_requests:error_rate5m > 5
  for: 5m

3. 注解模板

yaml 复制代码

- alert: HighMemory
  expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Memory usage > 90% on {{ $labels.instance }}"
    description: "Memory usage: {{ $value | printf \"%.1f\" }}%, available: {{ with printf \"node_memory_MemAvailable_bytes{instance=\\\"%s\\\"}\" $labels.instance | query }}{{ . | first | value | humanize1024 }}{{ end }}"
    dashboard: "https://grafana.example.com/d/node?var-instance={{ $labels.instance }}"
    runbook: "https://wiki.example.com/runbooks/high-memory"

4. 避免告警风暴

yaml 复制代码

# 使用 for 防止瞬时波动
- alert: PodCrashLooping
  expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
  for: 15m          # 持续 15 分钟才告警
  labels:
    severity: warning

# 使用 count 控制批量告警
- alert: ManyServicesDown
  expr: count(up == 0) > 3    # 超过 3 个服务宕机才告警
  for: 2m
  labels:
    severity: critical

四、完整告警规则集

yaml 复制代码

# rules/alerting_rules.yml
groups:
  # ===== 基础设施告警 =====
  - name: infrastructure
    rules:
      # 主机宕机
      - alert: NodeDown
        expr: up{job="node-exporter"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Node {{ $labels.instance }} is down"

      # CPU 高
      - alert: HighCPU
        expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "CPU > 80% on {{ $labels.instance }}"

      # 内存高
      - alert: HighMemory
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Memory > 90% on {{ $labels.instance }}"

      # 磁盘空间不足
      - alert: DiskSpaceLow
        expr: (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Disk > 85% on {{ $labels.instance }} {{ $labels.mountpoint }}"

      # 磁盘满预测
      - alert: DiskWillFillIn4h
        expr: predict_linear(node_filesystem_avail_bytes[1h], 4*3600) < 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Disk will fill in 4h on {{ $labels.instance }}"

      # 系统负载高
      - alert: HighLoad
        expr: node_load1 > count by(instance)(node_cpu_seconds_total{mode="idle"}) * 1.5
        for: 10m
        labels:
          severity: warning

      # MTR（Mean Time to Recover）监控
      - alert: NetworkErrors
        expr: rate(node_network_receive_errs_total[5m]) > 0
        for: 5m
        labels:
          severity: warning

  # ===== Kubernetes 告警 =====
  - name: kubernetes
    rules:
      # Pod CrashLoopBackOff
      - alert: PodCrashLooping
        expr: increase(kube_pod_container_status_restarts_total[15m]) > 5
        for: 5m
        labels:
          severity: warning

      # Pod Not Ready
      - alert: PodNotReady
        expr: kube_pod_status_ready{condition="true"} == 0
        for: 10m
        labels:
          severity: warning

      # Deployment 副本不足
      - alert: DeploymentReplicasMismatch
        expr: kube_deployment_spec_replicas != kube_deployment_status_replicas
        for: 10m
        labels:
          severity: warning

      # Node Not Ready
      - alert: KubernetesNodeNotReady
        expr: kube_node_status_condition{condition="Ready",status!="true"} == 1
        for: 5m
        labels:
          severity: critical

  # ===== 应用告警 =====
  - name: application
    rules:
      # 高错误率
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
          / sum(rate(http_requests_total[5m])) by (service) * 100 > 5
        for: 5m
        labels:
          severity: critical

      # 高延迟 P99
      - alert: HighLatency
        expr: |
          histogram_quantile(0.99,
            sum by (service, le)(rate(http_request_duration_seconds_bucket[5m]))) > 1
        for: 5m
        labels:
          severity: warning

      # 服务不可达
      - alert: ServiceUnavailable
        expr: up{job="app"} == 0
        for: 2m
        labels:
          severity: critical

  # ===== 数据库告警 =====
  - name: database
    rules:
      - alert: MySQLDown
        expr: mysql_up == 0
        for: 2m
        labels:
          severity: critical

      - alert: MySQLReplicationLag
        expr: mysql_slave_status_seconds_behind_master > 300
        for: 5m
        labels:
          severity: warning

      - alert: MySQLTooManyConnections
        expr: mysql_global_status_threads_connected / mysql_global_variables_max_connections * 100 > 80
        for: 5m
        labels:
          severity: warning

      - alert: RedisDown
        expr: redis_up == 0
        for: 2m
        labels:
          severity: critical

      - alert: RedisMemoryFull
        expr: redis_memory_used_bytes / redis_memory_max_bytes * 100 > 90
        for: 5m
        labels:
          severity: critical

五、配置与管理

Prometheus 配置

yaml 复制代码

# prometheus.yml
rule_files:
  - "rules/recording/*.yml"
  - "rules/alerting/*.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

验证规则

bash 复制代码

# 检查规则语法
promtool check rules rules/recording/*.yml rules/alerting/*.yml

# 检查配置
promtool check config prometheus.yml

# 查看已加载的规则
curl http://localhost:9090/api/v1/rules | jq

# 查看当前告警
curl http://localhost:9090/api/v1/alerts | jq

热重载

bash 复制代码

# 修改规则文件后热重载
curl -X POST http://localhost:9090/-/reload

⚠️ 踩坑提示 ：

告警名称必须全局唯一

for 时间不要太短，推荐至少 1-5 分钟

避免 for: 0，瞬时告警容易误报

Recording Rule 指标名不要和原始指标名冲突

每条规则都要写 annotations，方便接手人理解

要点回顾

规则类型	作用	使用场景
Recording Rules	预计算存储指标	复杂查询、频繁查询
Alerting Rules	触发告警	所有需要告警的场景

Recording Rule 命名：<scope>:<metric>:<operation>
Alerting Rule 三阶段：Pending → Firing → Resolved
for 防瞬时波动，至少 1-5 分钟
告警分级：critical / warning / info
annotations 写清 summary + description + runbook
验证用 promtool check rules

下一篇预告

告警规则搞定了，下一篇 【Prometheus·可视化篇】Grafana 集成：数据源配置与 Dashboard 设计原则 将讲解如何用 Grafana 做可视化。