XXL-Job 全链路监控看板和告警方案：从 Pod 健康到任务尾延迟的可观测闭环

meijinmeng2026-07-30 11:10

项目说明

复制代码 
# XXL-Job Exporter 监控

通过 [sql_exporter](https://github.com/burningalchemist/sql_exporter) 直查 XXL-Job RDS 数据库，将任务成功/失败数暴露为 Prometheus 指标，配合 Grafana 看板展示多环境 XXL-Job 运行状态。

## 目录结构

```
xxljob-exporter/
├── grafana-dashboard.json       # Grafana 看板（导入一次，所有环境共用）
├── qa/
│   ├── xxljob-sql-exporter.yaml # QA 环境：ConfigMap + Deployment + Service + ServiceMonitor
│   └── argocd-app.yaml          # QA 环境 ArgoCD Application
├── uat/                         # UAT 环境（参考 qa/ 新建）
├── prod/                        # Prod 环境（参考 qa/ 新建）
└── README.md
```

## 架构与映射原理

```
┌─────────────────────────────────────────────────────────────┐
│  用户在 Grafana 选择: xxljob_cluster = xxljob-qa            │
│                         ↓                                   │
│  隐藏变量自动推导:                                           │
│  $k8s_cluster = label_values(                               │
│    up{xxljob_cluster="xxljob-qa"}, cluster                  │
│  )  →  "ltp-nonprod-eks"                                    │
│                         ↓                                   │
│  sql-exporter 指标: xxljob_cluster="xxljob-qa"              │
│  kube-state-metrics 指标: cluster="ltp-nonprod-eks"         │
└─────────────────────────────────────────────────────────────┘
```

**映射关系不在 dashboard 里维护**，由两处配置自动决定：

| 配置项 | 位置 | 作用 |
|--------|------|------|
| `externalLabels.cluster` | 各集群 Prometheus values.yaml | 决定 `$k8s_cluster` 的值 |
| ServiceMonitor `xxljob_cluster` relabeling | 各环境 `xxljob-sql-exporter.yaml` | 决定 `$xxljob_cluster` 的值 |

**当前环境映射：**

| xxljob_cluster | k8s cluster | 说明 |
|----------------|-------------|------|
| xxljob-qa | ltp-nonprod-eks | QA 环境 |
| xxljob-prod | ltp-eks-prod | 生产环境 |

## 新增环境（以 UAT 为例）

### Step 1 --- 复制 QA 配置

```bash
cp -r xxljob-exporter/qa xxljob-exporter/uat
```

### Step 2 — 修改 `uat/xxljob-sql-exporter.yaml`

需要修改以下字段：

```yaml
# 1. target --- RDS 地址改为 UAT 的
target:
  data_source_name: "mysql://root:${DB_PASSWORD}@<uat-rds-endpoint>:3306/xxl_job?..."

# 2. ServiceMonitor relabelings --- 改为 UAT 的标识
relabelings:
  - targetLabel: xxljob_cluster
    replacement: xxljob-uat          # ← 改这里，全局唯一
  - targetLabel: xxljob_dns
    replacement: https://uat-xxljob.icbc.com/xxl-job-admin/  # ← 改这里
```

### Step 3 — 修改 `uat/argocd-app.yaml`

```yaml
metadata:
  name: xxljob-exporter-uat          # ← 改名

spec:
  source:
    path: xxljob-exporter/uat        # ← 改路径
  destination:
    server: https://<uat-eks-api-endpoint>   # ← UAT 集群的 API server
    namespace: xxl-job
```

### Step 4 --- 创建 DB Secret

在 UAT 集群执行（Secret 不纳入 Git）：

```bash
kubectl create secret generic xxljob-db-secret \
  --from-literal=db-password='<uat-db-password>' \
  -n xxl-job \
  --context <uat-kubeconfig-context>
```

### Step 5 --- 提交并部署

```bash
git add xxljob-exporter/uat/
git commit -m "feat: add xxljob exporter for uat"
git push

# 在 UAT 集群 ArgoCD 中接管
kubectl apply -f xxljob-exporter/uat/argocd-app.yaml -n argocd
```

### Step 6 --- 确认映射生效

部署后约 2 分钟，在 Prometheus 中验证：

```promql
up{xxljob_cluster="xxljob-uat"}
```

返回结果中应同时包含 `cluster="ltp-uat-eks"`（或对应集群名）。

Grafana 看板下拉菜单刷新后自动出现 `xxljob-uat` 选项，**无需修改 dashboard**。

## Grafana 看板导入

1. Grafana → Dashboards → Import
2. 上传 `grafana-dashboard.json`
3. 选择对应的 Prometheus datasource
4. 导入后所有环境共用此看板

看板变量说明：

| 变量 | 类型 | 说明 |
|------|------|------|
| `$datasource` | 数据源 | Prometheus 实例 |
| `$xxljob_cluster` | 下拉选择 | XXL-Job 集群标识，用户可见 |
| `$k8s_cluster` | 隐藏 | 自动从 `up` 指标映射出 K8s 集群名，用于过滤 kube 指标 |
| `$xxljob_dns` | 下拉选择 | 管理后台 URL，随 `$xxljob_cluster` 联动 |
| `$namespace` | 下拉选择 | 随 `$k8s_cluster` 过滤后的命名空间 |

## 指标说明

| 指标名 | 类型 | 说明 | 采集周期 |
|--------|------|------|----------|
| `xxl_job_task_result_success_count` | gauge | 近 1h 各业务组成功数 | 120s |
| `xxl_job_task_result_fail_count` | gauge | 近 1h 各业务组失败数 | 120s |
| `xxl_job_task_result_running_count` | gauge | 近 1h 各业务组运行中数 | 120s |
| `xxl_job_task_daily_success_count` | gauge | 近 24h 各 Job 成功数 | 300s |
| `xxl_job_task_daily_fail_count` | gauge | 近 24h 各 Job 失败数 | 300s |

## 故障排查

**下拉菜单集群/管理后台为空**

```promql
# 验证 up 指标是否携带 xxljob_cluster 标签
up{xxljob_cluster=~".+"}
```

若无结果，检查：
1. sql-exporter pod 是否运行：`kubectl get pods -n xxl-job`
2. Prometheus Targets 页面是否能发现该 exporter
3. ServiceMonitor namespace 与 Prometheus 是否在同一集群

**Pod 数量显示不正确（偏多）**

多个集群的 kube-state-metrics 指标汇聚到同一 Prometheus 时会出现重复计数。
看板已通过 `cluster="$k8s_cluster"` + `max by (namespace, pod)` 处理去重，
若仍有问题确认 `$k8s_cluster` 变量是否正确解析。

**SQL 查询无数据**

确认 RDS Security Group 允许 EKS Node → 3306，并验证 DB 密码 Secret 存在：

```bash
kubectl get secret xxljob-db-secret -n xxl-job
```

View Code

功能说明

复制代码 
1. 服务健康监控
Pod 状态：UP/DOWN 状态指示，一眼判断 Admin 是否存活
重启次数（1h）：阈值告警（0→绿，1→黄，5→红），快速发现 CrashLoop
Pod 运行时长：判断是否近期被重启过
Ready Pod 数量：多副本场景下反映可用实例数
CPU / Memory Gauge：当前资源占用与上限对比，超限风险可视
2. 资源使用趋势
CPU 使用量：实际使用 vs Limit 两条线叠加，资源争抢一目了然
内存使用量：同上，OOM 风险监控
3. 任务总览（1h + 24h 双时间窗口）
整体成功率 Gauge：近 1h 和近 24h 各一个，短期波动 + 长期趋势对比
成功/失败/运行中任务数：核心 KPI 指标卡
总执行数 / 整体失败率：吞吐量与质量双视角
4. 任务执行趋势
成功数（绿）、失败数（红）、运行中 三条曲线，每分钟采样
失败曲线有 hideSeriesFrom 配置，默认隐藏失败数，聚焦成功趋势时仍可切换查看
5. 任务执行耗时分位数 ✨ 新增
P95 / P99 执行耗时：反映长尾任务的执行时长分布
区分"成功任务的耗时分布"与"整体延迟水位"，支撑慢任务排查
与成功/失败计数互补，从"量"（做了多少）扩展到"质"（做得多慢）
6. 业务组详情
各业务组失败趋势：仅展示 > 0 的失败组，过滤噪音
业务组明细表：成功数/失败数/成功率三列，按失败数排序，带渐变背景色，快速定位最差业务组
7. 24h Job 明细 ✨ 优化
粒度细化到 job_name 级别（业务组 + 任务名称）
Top N 限制 / 分页：大集群下 job_name 众多时，只展示最值得关注的部分（如失败数 Top N 或耗时 Top N），避免行数爆炸、表格卡顿
按失败数排序，成功率渐变着色
8. 模板变量（联动筛选）
变量    说明
datasource    数据源选择
xxljob_cluster    XXL-Job 集群（主变量）
k8s_cluster    自动联动 K8s 集群（hide=1，隐藏不可见）
xxljob_dns    管理后台地址，可一键跳转
namespace    命名空间
整体刷新 5 分钟，时间默认近 3 小时

项目实现

复制代码 
# =============================================================================
# XXL-Job SQL Exporter --- QA 环境
# =============================================================================
# 通过直查 RDS xxl_job_log 表，将任务成功/失败数暴露为 Prometheus 指标
#
# 依赖：
#   - xxl-job namespace 已存在（或 ArgoCD syncOptions 开启 CreateNamespace）
#   - RDS Security Group 允许 EKS Node → 3306
#   - Prometheus Operator 已部署（ServiceMonitor CRD 存在）
#
# Secret 说明：
#   DB 密码当前存于 xxljob-db-secret，请勿将真实密码提交 Git。
#   生产环境建议替换为 SealedSecret 或 External Secrets Operator (AWS Secrets Manager)。
#
# 更新 SQL 查询：
#   修改下方 ConfigMap data.config.yml 中的 query 字段，ArgoCD Sync 后自动生效。
# =============================================================================

# Secret xxljob-db-secret 通过 kubectl create secret 手动创建，不纳入 Git 管理。
# kubectl create secret generic xxljob-db-secret --from-literal=db-password='xxx' -n xxl-job

# -----------------------------------------------------------------------------
# ConfigMap --- sql_exporter 查询配置
# -----------------------------------------------------------------------------
apiVersion: v1
kind: ConfigMap
metadata:
  name: xxljob-sql-exporter-config
  namespace: monitoring
data:
  config.yml: |
    global:
      scrape_timeout: 50s
      scrape_timeout_offset: 500ms
      max_connections: 6
      max_idle_connections: 3

    target:
      data_source_name: "mysql://xxl-job:${DB_PASSWORD}@hostname:3306/xxl_job?parseTime=true&loc=UTC"
      collectors: [xxl_job_metrics_1h, xxl_job_metrics_24h, xxl_job_duration_1h]

    collectors:
      # 近 1 小时：按业务组统计，每 60s 采集一次
      - collector_name: xxl_job_metrics_1h
        min_interval: 120s
        metrics:
          - metric_name: xxl_job_task_result_success_count
            type: gauge
            help: "XXL-Job success count by group (last 1h)"
            key_labels: [job_group_name]
            values: [success_count]
            query_ref: task_result_1h

          - metric_name: xxl_job_task_result_fail_count
            type: gauge
            help: "XXL-Job fail count by group (last 1h)"
            key_labels: [job_group_name]
            values: [fail_count]
            query_ref: task_result_1h

          - metric_name: xxl_job_task_result_running_count
            type: gauge
            help: "XXL-Job running count by group (last 1h)"
            key_labels: [job_group_name]
            values: [running_count]
            query_ref: task_result_1h

        queries:
          - query_name: task_result_1h
            query: |
              SELECT
                COALESCE(g.title, CAST(l.job_group AS CHAR)) AS job_group_name,
                COUNT(CASE WHEN l.handle_code = 200 THEN 1 END)                        AS success_count,
                COUNT(CASE WHEN l.handle_code IN (500, 502) THEN 1 END)                AS fail_count,
                COUNT(CASE WHEN l.handle_code = 0 AND l.trigger_code = 200 THEN 1 END) AS running_count
              FROM xxl_job_log l
              LEFT JOIN xxl_job_group g ON l.job_group = g.id
              WHERE l.trigger_time >= DATE_SUB(NOW(), INTERVAL 1 HOUR)
              GROUP BY l.job_group, g.title

      # 近 24 小时：按具体 Job 统计，每 5m 采集一次降低全表扫描压力
      - collector_name: xxl_job_metrics_24h
        min_interval: 300s
        metrics:
          - metric_name: xxl_job_task_daily_success_count
            type: gauge
            help: "XXL-Job success count per job (last 24h)"
            key_labels: [job_group_name, job_name]
            values: [success_count]
            query_ref: task_daily_24h

          - metric_name: xxl_job_task_daily_fail_count
            type: gauge
            help: "XXL-Job fail count per job (last 24h)"
            key_labels: [job_group_name, job_name]
            values: [fail_count]
            query_ref: task_daily_24h

        queries:
          - query_name: task_daily_24h
            query: |
              SELECT
                COALESCE(g.title, CAST(l.job_group AS CHAR)) AS job_group_name,
                COALESCE(i.job_desc, CAST(l.job_id AS CHAR)) AS job_name,
                COUNT(CASE WHEN l.handle_code = 200 THEN 1 END)         AS success_count,
                COUNT(CASE WHEN l.handle_code IN (500, 502) THEN 1 END) AS fail_count
              FROM xxl_job_log l
              LEFT JOIN xxl_job_group g ON l.job_group = g.id
              LEFT JOIN xxl_job_info  i ON l.job_id   = i.id
              WHERE l.trigger_time >= DATE_SUB(NOW(), INTERVAL 24 HOUR)
              GROUP BY l.job_group, g.title, l.job_id, i.job_desc
              ORDER BY fail_count DESC, success_count DESC
              LIMIT 50

      # 执行耗时：avg/max/慢任务数，每 2m 采集，用于排查慢任务
      - collector_name: xxl_job_duration_1h
        min_interval: 120s
        metrics:
          - metric_name: xxl_job_task_avg_duration_seconds
            type: gauge
            help: "XXL-Job avg execution duration per group (last 1h, success only)"
            key_labels: [job_group_name]
            values: [avg_duration_s]
            query_ref: duration_1h

          - metric_name: xxl_job_task_max_duration_seconds
            type: gauge
            help: "XXL-Job max execution duration per group (last 1h, success only)"
            key_labels: [job_group_name]
            values: [max_duration_s]
            query_ref: duration_1h

          - metric_name: xxl_job_task_slow_count
            type: gauge
            help: "XXL-Job tasks exceeding 60s per group (last 1h)"
            key_labels: [job_group_name]
            values: [slow_gt60s]
            query_ref: duration_1h

        queries:
          - query_name: duration_1h
            query: |
              SELECT
                COALESCE(g.title, CAST(l.job_group AS CHAR)) AS job_group_name,
                ROUND(AVG(TIMESTAMPDIFF(SECOND, l.trigger_time, l.handle_time)), 1) AS avg_duration_s,
                MAX(TIMESTAMPDIFF(SECOND, l.trigger_time, l.handle_time))           AS max_duration_s,
                SUM(CASE WHEN TIMESTAMPDIFF(SECOND, l.trigger_time, l.handle_time) > 60 THEN 1 ELSE 0 END) AS slow_gt60s
              FROM xxl_job_log l
              LEFT JOIN xxl_job_group g ON l.job_group = g.id
              WHERE l.trigger_time >= DATE_SUB(NOW(), INTERVAL 1 HOUR)
                AND l.handle_code = 200
                AND l.handle_time IS NOT NULL
                AND l.trigger_time IS NOT NULL
              GROUP BY l.job_group, g.title

---
# -----------------------------------------------------------------------------
# Deployment --- sql_exporter
# -----------------------------------------------------------------------------
apiVersion: apps/v1
kind: Deployment
metadata:
  name: xxljob-sql-exporter
  namespace: monitoring
  labels:
    app: xxljob-sql-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: xxljob-sql-exporter
  template:
    metadata:
      labels:
        app: xxljob-sql-exporter
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9399"
        prometheus.io/path: "/metrics"
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
      containers:
        - name: sql-exporter
          image: burningalchemist/sql_exporter:0.14.1
          imagePullPolicy: IfNotPresent
          args:
            - -config.file=/config/config.yml
          ports:
            - containerPort: 9399
              name: metrics
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: xxljob-db-secret
                  key: db-password
          resources:
            requests:
              cpu: 50m
              memory: 128Mi
            limits:
              cpu: 200m
              memory: 512Mi
          readinessProbe:
            httpGet:
              path: /healthz
              port: 9399
            initialDelaySeconds: 10
            periodSeconds: 30
          livenessProbe:
            httpGet:
              path: /healthz
              port: 9399
            initialDelaySeconds: 30
            periodSeconds: 60
          volumeMounts:
            - name: config
              mountPath: /config
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: xxljob-sql-exporter-config

---
# -----------------------------------------------------------------------------
# Service --- 暴露 metrics 端口
# -----------------------------------------------------------------------------
apiVersion: v1
kind: Service
metadata:
  name: xxljob-sql-exporter
  namespace: monitoring
  labels:
    app: xxljob-sql-exporter
spec:
  selector:
    app: xxljob-sql-exporter
  ports:
    - port: 9399
      targetPort: 9399
      name: metrics

---
# -----------------------------------------------------------------------------
# ServiceMonitor --- Prometheus Operator 自动发现
# -----------------------------------------------------------------------------
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: xxljob-sql-exporter
  namespace: monitoring
  labels:
    release: kube-prom-stack   # ← 改成你的 Prometheus Operator release label
spec:
  selector:
    matchLabels:
      app: xxljob-sql-exporter
  namespaceSelector:
    matchNames:
      - monitoring
  endpoints:
    - port: metrics
      interval: 60s
      scrapeTimeout: 58s
      relabelings:
        - targetLabel: xxljob_cluster
          replacement: xxljob-prod
        - targetLabel: xxljob_dns
          replacement: https://xxljob.icbc.com/xxl-job-admin/


---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: xxljob-exporter-prod
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/icbc/devops-deploy.git
    targetRevision: main
    path: xxljob-exporter/prod
    directory:
      exclude: 'argocd-app.yaml'
  destination:
    server: https://icbc.eks.amazonaws.com
    namespace: monitoring

  # 首次接管步骤：
  #   1. kubectl apply -f argocd-app.yaml -n argocd
  #   2. ArgoCD UI → xxljob-exporter-qa → Refresh → 确认 diff 无误
  #   3. 手动点 Sync 或开启下方 automated
  syncPolicy:
    # automated:
    #   prune: false
    #   selfHeal: false
    retry:
      limit: 3
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
    syncOptions:
      - ApplyOutOfSyncOnly=true
      - ServerSideApply=true
      - CreateNamespace=true

deploy & argocd-app

Dashboard监控看板

复制代码 
{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana",
          "uid": "-- Grafana --"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "description": "XXL-Job Admin 监控看板 --- Pod 健康、资源使用、任务成功/失败率",
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 1,
  "id": 515,
  "links": [],
  "panels": [
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "id": 100,
      "panels": [],
      "title": "服务健康",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [
            {
              "options": {
                "0": {
                  "color": "red",
                  "index": 0,
                  "text": "DOWN"
                },
                "1": {
                  "color": "green",
                  "index": 1,
                  "text": "UP"
                }
              },
              "type": "value"
            }
          ],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "red",
                "value": 0
              },
              {
                "color": "green",
                "value": 1
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 0,
        "y": 1
      },
      "id": 1,
      "options": {
        "colorMode": "background",
        "graphMode": "none",
        "justifyMode": "center",
        "orientation": "auto",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "auto",
        "wideLayout": true
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "max(kube_pod_status_ready{namespace=\"$namespace\", cluster=\"$k8s_cluster\", pod=~\"xxl-job-admin.*\", condition=\"true\"})",
          "legendFormat": "Admin Pod",
          "refId": "A"
        }
      ],
      "title": "Admin Pod 状态",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "yellow",
                "value": 1
              },
              {
                "color": "red",
                "value": 5
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 4,
        "y": 1
      },
      "id": 2,
      "options": {
        "colorMode": "background",
        "graphMode": "none",
        "justifyMode": "center",
        "orientation": "auto",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "auto",
        "wideLayout": true
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(max by (namespace, pod, container) (increase(kube_pod_container_status_restarts_total{namespace=\"$namespace\", cluster=\"$k8s_cluster\", container=\"xxl-job-admin\"}[1h])))",
          "legendFormat": "重启次数(1h)",
          "refId": "A"
        }
      ],
      "title": "Pod 重启次数 (1h)",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "blue",
                "value": 0
              }
            ]
          },
          "unit": "s"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 8,
        "y": 1
      },
      "id": 3,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "center",
        "orientation": "auto",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "auto",
        "wideLayout": true
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "time() - max(kube_pod_start_time{namespace=\"$namespace\", cluster=\"$k8s_cluster\", pod=~\"xxl-job-admin.*\"})",
          "legendFormat": "运行时长",
          "refId": "A"
        }
      ],
      "title": "Pod 运行时长",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "green",
                "value": 1
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 12,
        "y": 1
      },
      "id": 4,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "center",
        "orientation": "auto",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "auto",
        "wideLayout": true
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "count(max by (namespace, pod) (kube_pod_status_ready{namespace=\"$namespace\", cluster=\"$k8s_cluster\", pod=~\"xxl-job-admin.*\", condition=\"true\"}) == 1)",
          "legendFormat": "Ready Pods",
          "refId": "A"
        }
      ],
      "title": "Ready Pod 数量",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "decimals": 3,
          "mappings": [],
          "max": 2,
          "min": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "yellow",
                "value": 0.5
              },
              {
                "color": "red",
                "value": 1
              }
            ]
          },
          "unit": "none"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 5,
        "w": 6,
        "x": 0,
        "y": 5
      },
      "id": 19,
      "options": {
        "minVizHeight": 75,
        "minVizWidth": 75,
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true,
        "sizing": "auto",
        "text": {}
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"$namespace\", cluster=\"$k8s_cluster\", container=\"xxl-job-admin\", image!=\"\"}[5m]))",
          "legendFormat": "CPU (cores)",
          "refId": "A"
        }
      ],
      "title": "CPU 使用量 (cores)",
      "type": "gauge"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "max": 2147483648,
          "min": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "yellow",
                "value": 1073741824
              },
              {
                "color": "red",
                "value": 1610612736
              }
            ]
          },
          "unit": "bytes"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 5,
        "w": 6,
        "x": 6,
        "y": 5
      },
      "id": 20,
      "options": {
        "minVizHeight": 75,
        "minVizWidth": 75,
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true,
        "sizing": "auto",
        "text": {}
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(container_memory_working_set_bytes{namespace=\"$namespace\", cluster=\"$k8s_cluster\", container=\"xxl-job-admin\", image!=\"\"})",
          "legendFormat": "Memory",
          "refId": "A"
        }
      ],
      "title": "Memory 使用量",
      "type": "gauge"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 10
      },
      "id": 101,
      "panels": [],
      "title": "资源使用",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 2,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "none"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 7,
        "w": 12,
        "x": 0,
        "y": 11
      },
      "id": 5,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "max",
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "multi",
          "sort": "none"
        }
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"$namespace\", cluster=\"$k8s_cluster\", container=\"xxl-job-admin\", image!=\"\"}[5m]))",
          "legendFormat": "CPU Usage",
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(kube_pod_container_resource_limits{namespace=\"$namespace\", cluster=\"$k8s_cluster\", container=\"xxl-job-admin\", resource=\"cpu\"})",
          "legendFormat": "CPU Limit",
          "refId": "B"
        }
      ],
      "title": "CPU 使用量 (cores)",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 2,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "bytes"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 7,
        "w": 12,
        "x": 12,
        "y": 11
      },
      "id": 6,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "max",
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "multi",
          "sort": "none"
        }
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(container_memory_working_set_bytes{namespace=\"$namespace\", cluster=\"$k8s_cluster\", container=\"xxl-job-admin\", image!=\"\"})",
          "legendFormat": "Memory Usage",
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(kube_pod_container_resource_limits{namespace=\"$namespace\", cluster=\"$k8s_cluster\", container=\"xxl-job-admin\", resource=\"memory\"})",
          "legendFormat": "Memory Limit",
          "refId": "B"
        }
      ],
      "title": "内存使用量",
      "type": "timeseries"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 18
      },
      "id": 102,
      "panels": [],
      "title": "任务总览 (近 1 小时)",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "max": 100,
          "min": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "red",
                "value": 0
              },
              {
                "color": "yellow",
                "value": 80
              },
              {
                "color": "green",
                "value": 95
              }
            ]
          },
          "unit": "percent"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 6,
        "w": 6,
        "x": 0,
        "y": 19
      },
      "id": 7,
      "options": {
        "minVizHeight": 75,
        "minVizWidth": 75,
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true,
        "sizing": "auto"
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(xxl_job_task_result_success_count{xxljob_cluster=\"$xxljob_cluster\"}) / clamp_min(sum(xxl_job_task_result_success_count{xxljob_cluster=\"$xxljob_cluster\"}) + sum(xxl_job_task_result_fail_count{xxljob_cluster=\"$xxljob_cluster\"}), 1) * 100",
          "legendFormat": "成功率",
          "refId": "A"
        }
      ],
      "title": "整体任务成功率 (近 1h)",
      "type": "gauge"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 4,
        "x": 6,
        "y": 19
      },
      "id": 8,
      "options": {
        "colorMode": "value",
        "graphMode": "area",
        "justifyMode": "center",
        "orientation": "auto",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "auto",
        "wideLayout": true
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(xxl_job_task_result_success_count{xxljob_cluster=\"$xxljob_cluster\"})",
          "legendFormat": "成功数",
          "refId": "A"
        }
      ],
      "title": "成功任务数",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "red",
                "value": 1
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 4,
        "x": 10,
        "y": 19
      },
      "id": 9,
      "options": {
        "colorMode": "background",
        "graphMode": "area",
        "justifyMode": "center",
        "orientation": "auto",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "auto",
        "wideLayout": true
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(xxl_job_task_result_fail_count{xxljob_cluster=\"$xxljob_cluster\"})",
          "legendFormat": "失败数",
          "refId": "A"
        }
      ],
      "title": "失败任务数",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "yellow",
                "value": 10
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 4,
        "x": 14,
        "y": 19
      },
      "id": 10,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "center",
        "orientation": "auto",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "auto",
        "wideLayout": true
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(xxl_job_task_result_running_count{xxljob_cluster=\"$xxljob_cluster\"})",
          "legendFormat": "运行中",
          "refId": "A"
        }
      ],
      "title": "运行中任务数",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "max": 100,
          "min": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "red",
                "value": 0
              },
              {
                "color": "yellow",
                "value": 80
              },
              {
                "color": "green",
                "value": 95
              }
            ]
          },
          "unit": "percent"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 6,
        "w": 6,
        "x": 18,
        "y": 19
      },
      "id": 18,
      "options": {
        "minVizHeight": 75,
        "minVizWidth": 75,
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true,
        "sizing": "auto"
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(xxl_job_task_daily_success_count{xxljob_cluster=\"$xxljob_cluster\"}) / clamp_min(sum(xxl_job_task_daily_success_count{xxljob_cluster=\"$xxljob_cluster\"}) + sum(xxl_job_task_daily_fail_count{xxljob_cluster=\"$xxljob_cluster\"}), 1) * 100",
          "legendFormat": "24h成功率",
          "refId": "A"
        }
      ],
      "title": "整体任务成功率 (近 24h)",
      "type": "gauge"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "blue",
                "value": 0
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 4,
        "x": 6,
        "y": 22
      },
      "id": 11,
      "options": {
        "colorMode": "value",
        "graphMode": "area",
        "justifyMode": "center",
        "orientation": "auto",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "auto",
        "wideLayout": true
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(xxl_job_task_result_success_count{xxljob_cluster=\"$xxljob_cluster\"}) + sum(xxl_job_task_result_fail_count{xxljob_cluster=\"$xxljob_cluster\"})",
          "legendFormat": "总执行数",
          "refId": "A"
        }
      ],
      "title": "总执行数",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "max": 100,
          "min": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "yellow",
                "value": 5
              },
              {
                "color": "red",
                "value": 10
              }
            ]
          },
          "unit": "percent"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 4,
        "x": 10,
        "y": 22
      },
      "id": 12,
      "options": {
        "colorMode": "background",
        "graphMode": "none",
        "justifyMode": "center",
        "orientation": "auto",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "auto",
        "wideLayout": true
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(xxl_job_task_result_fail_count{xxljob_cluster=\"$xxljob_cluster\"}) / clamp_min(sum(xxl_job_task_result_success_count{xxljob_cluster=\"$xxljob_cluster\"}) + sum(xxl_job_task_result_fail_count{xxljob_cluster=\"$xxljob_cluster\"}), 1) * 100",
          "legendFormat": "失败率",
          "refId": "A"
        }
      ],
      "title": "整体失败率",
      "type": "stat"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 25
      },
      "id": 103,
      "panels": [],
      "title": "任务趋势",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 15,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 2,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "short"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "失败数"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "red",
                  "mode": "fixed"
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "成功数"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "green",
                  "mode": "fixed"
                }
              }
            ]
          },
          {
            "__systemRef": "hideSeriesFrom",
            "matcher": {
              "id": "byNames",
              "options": {
                "mode": "exclude",
                "names": [
                  "失败数"
                ],
                "prefix": "All except:",
                "readOnly": true
              }
            },
            "properties": [
              {
                "id": "custom.hideFrom",
                "value": {
                  "legend": false,
                  "tooltip": false,
                  "viz": true
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 8,
        "w": 24,
        "x": 0,
        "y": 26
      },
      "id": 13,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "max",
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "right",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "multi",
          "sort": "none"
        }
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(xxl_job_task_result_success_count{xxljob_cluster=\"$xxljob_cluster\"})",
          "interval": "60s",
          "legendFormat": "成功数",
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(xxl_job_task_result_fail_count{xxljob_cluster=\"$xxljob_cluster\"})",
          "interval": "60s",
          "legendFormat": "失败数",
          "refId": "B"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(xxl_job_task_result_running_count{xxljob_cluster=\"$xxljob_cluster\"})",
          "interval": "60s",
          "legendFormat": "运行中",
          "refId": "C"
        }
      ],
      "title": "任务执行趋势 (每分钟采样)",
      "type": "timeseries"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 34
      },
      "id": 104,
      "panels": [],
      "title": "业务组详情",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 2,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 35
      },
      "id": 14,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "max",
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "multi",
          "sort": "none"
        }
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "xxl_job_task_result_fail_count{xxljob_cluster=\"$xxljob_cluster\"} > 0",
          "legendFormat": "{{job_group_name}}",
          "refId": "A"
        }
      ],
      "title": "各业务组失败趋势 (近 1h)",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "custom": {
            "align": "auto",
            "cellOptions": {
              "type": "auto"
            },
            "inspect": false
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "short"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "成功率"
            },
            "properties": [
              {
                "id": "unit",
                "value": "percent"
              },
              {
                "id": "custom.cellOptions",
                "value": {
                  "mode": "gradient",
                  "type": "color-background"
                }
              },
              {
                "id": "thresholds",
                "value": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "red",
                      "value": 0
                    },
                    {
                      "color": "yellow",
                      "value": 80
                    },
                    {
                      "color": "green",
                      "value": 95
                    }
                  ]
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "失败数"
            },
            "properties": [
              {
                "id": "custom.cellOptions",
                "value": {
                  "mode": "gradient",
                  "type": "color-background"
                }
              },
              {
                "id": "thresholds",
                "value": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "green",
                      "value": 0
                    },
                    {
                      "color": "yellow",
                      "value": 1
                    },
                    {
                      "color": "red",
                      "value": 5
                    }
                  ]
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 35
      },
      "id": 15,
      "options": {
        "cellHeight": "sm",
        "footer": {
          "countRows": false,
          "fields": "",
          "reducer": [
            "sum"
          ],
          "show": false
        },
        "showHeader": true,
        "sortBy": [
          {
            "desc": true,
            "displayName": "失败数"
          }
        ]
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "xxl_job_task_result_success_count{xxljob_cluster=\"$xxljob_cluster\"}",
          "format": "table",
          "instant": true,
          "legendFormat": "",
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "xxl_job_task_result_fail_count{xxljob_cluster=\"$xxljob_cluster\"}",
          "format": "table",
          "instant": true,
          "legendFormat": "",
          "refId": "B"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "xxl_job_task_result_success_count{xxljob_cluster=\"$xxljob_cluster\"} / clamp_min(xxl_job_task_result_success_count{xxljob_cluster=\"$xxljob_cluster\"} + xxl_job_task_result_fail_count{xxljob_cluster=\"$xxljob_cluster\"}, 1) * 100",
          "format": "table",
          "instant": true,
          "legendFormat": "",
          "refId": "C"
        }
      ],
      "title": "业务组任务明细 (近 1h, 按失败数排序)",
      "transformations": [
        {
          "id": "merge",
          "options": {}
        },
        {
          "id": "filterFieldsByName",
          "options": {
            "include": {
              "pattern": "^(job_group_name|Value #[ABC])$"
            }
          }
        },
        {
          "id": "organize",
          "options": {
            "renameByName": {
              "Value #A": "成功数",
              "Value #B": "失败数",
              "Value #C": "成功率",
              "job_group_name": "业务组"
            }
          }
        }
      ],
      "type": "table"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 43
      },
      "id": 200,
      "panels": [],
      "title": "执行耗时分析",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "lineWidth": 2,
            "fillOpacity": 5
          },
          "unit": "s"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 44
      },
      "id": 201,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "max",
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "bottom"
        },
        "tooltip": {
          "mode": "multi"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "xxl_job_task_avg_duration_seconds{xxljob_cluster=\"$xxljob_cluster\"}",
          "legendFormat": "{{job_group_name}} avg",
          "refId": "A"
        }
      ],
      "title": "各业务组平均执行耗时 (近 1h)",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "lineWidth": 2,
            "fillOpacity": 5
          },
          "unit": "s"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 44
      },
      "id": 202,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "max",
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "bottom"
        },
        "tooltip": {
          "mode": "multi"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "xxl_job_task_max_duration_seconds{xxljob_cluster=\"$xxljob_cluster\"}",
          "legendFormat": "{{job_group_name}} max",
          "refId": "A"
        }
      ],
      "title": "各业务组最大执行耗时 (近 1h)",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "lineWidth": 2,
            "fillOpacity": 5
          },
          "unit": "percent"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 7,
        "w": 24,
        "x": 0,
        "y": 52
      },
      "id": 203,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "max",
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "bottom"
        },
        "tooltip": {
          "mode": "multi"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "xxl_job_task_slow_count{xxljob_cluster=\"$xxljob_cluster\"} / clamp_min(xxl_job_task_result_success_count{xxljob_cluster=\"$xxljob_cluster\"}, 1) * 100",
          "legendFormat": "{{job_group_name}}",
          "refId": "A"
        }
      ],
      "title": "慢任务率 >60s (近 1h)",
      "type": "timeseries"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 60
      },
      "id": 105,
      "panels": [],
      "title": "24小时 Job 明细",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "custom": {
            "align": "auto",
            "cellOptions": {
              "type": "auto"
            },
            "inspect": false
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "short"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "成功率(24h)"
            },
            "properties": [
              {
                "id": "unit",
                "value": "percent"
              },
              {
                "id": "custom.cellOptions",
                "value": {
                  "mode": "gradient",
                  "type": "color-background"
                }
              },
              {
                "id": "thresholds",
                "value": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "red",
                      "value": 0
                    },
                    {
                      "color": "yellow",
                      "value": 80
                    },
                    {
                      "color": "green",
                      "value": 95
                    }
                  ]
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "失败数(24h)"
            },
            "properties": [
              {
                "id": "custom.cellOptions",
                "value": {
                  "mode": "gradient",
                  "type": "color-background"
                }
              },
              {
                "id": "thresholds",
                "value": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "green",
                      "value": 0
                    },
                    {
                      "color": "yellow",
                      "value": 1
                    },
                    {
                      "color": "red",
                      "value": 5
                    }
                  ]
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 10,
        "w": 24,
        "x": 0,
        "y": 61
      },
      "id": 16,
      "options": {
        "cellHeight": "sm",
        "footer": {
          "countRows": false,
          "fields": "",
          "reducer": [
            "sum"
          ],
          "show": false
        },
        "showHeader": true,
        "sortBy": [
          {
            "desc": true,
            "displayName": "失败数(24h)"
          }
        ]
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "xxl_job_task_daily_success_count{xxljob_cluster=\"$xxljob_cluster\"}",
          "format": "table",
          "instant": true,
          "legendFormat": "",
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "xxl_job_task_daily_fail_count{xxljob_cluster=\"$xxljob_cluster\"}",
          "format": "table",
          "instant": true,
          "legendFormat": "",
          "refId": "B"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "xxl_job_task_daily_success_count{xxljob_cluster=\"$xxljob_cluster\"} / clamp_min(xxl_job_task_daily_success_count{xxljob_cluster=\"$xxljob_cluster\"} + xxl_job_task_daily_fail_count{xxljob_cluster=\"$xxljob_cluster\"}, 1) * 100",
          "format": "table",
          "instant": true,
          "legendFormat": "",
          "refId": "C"
        }
      ],
      "title": "24小时 Job 执行明细 Top 50 (按失败数排序)",
      "transformations": [
        {
          "id": "merge",
          "options": {}
        },
        {
          "id": "filterFieldsByName",
          "options": {
            "include": {
              "pattern": "^(job_group_name|job_name|Value #[ABC])$"
            }
          }
        },
        {
          "id": "organize",
          "options": {
            "renameByName": {
              "Value #A": "成功数(24h)",
              "Value #B": "失败数(24h)",
              "Value #C": "成功率(24h)",
              "job_group_name": "业务组",
              "job_name": "任务名称"
            }
          }
        }
      ],
      "type": "table"
    }
  ],
  "preload": false,
  "refresh": "5m",
  "schemaVersion": 41,
  "tags": [
    "xxl-job",
    "task-scheduler",
    "monitoring"
  ],
  "templating": {
    "list": [
      {
        "current": {
          "text": "Metrics",
          "value": "fetdvd0e806bkc"
        },
        "includeAll": false,
        "label": "数据源",
        "name": "datasource",
        "options": [],
        "query": "prometheus",
        "refresh": 1,
        "type": "datasource"
      },
      {
        "current": {
          "text": "xxljob-prod",
          "value": "xxljob-prod"
        },
        "datasource": {
          "type": "prometheus",
          "uid": "${datasource}"
        },
        "definition": "label_values(up{xxljob_cluster=~\".+\"}, xxljob_cluster)",
        "includeAll": false,
        "label": "集群",
        "name": "xxljob_cluster",
        "options": [],
        "query": {
          "query": "label_values(up{xxljob_cluster=~\".+\"}, xxljob_cluster)",
          "refId": "xxljob_cluster"
        },
        "refresh": 2,
        "sort": 1,
        "type": "query"
      },
      {
        "current": {
          "text": "ltp-eks-prod",
          "value": "ltp-eks-prod"
        },
        "datasource": {
          "type": "prometheus",
          "uid": "${datasource}"
        },
        "definition": "label_values(up{xxljob_cluster=\"$xxljob_cluster\"}, cluster)",
        "hide": 1,
        "includeAll": false,
        "label": "K8s集群(自动)",
        "name": "k8s_cluster",
        "options": [],
        "query": {
          "query": "label_values(up{xxljob_cluster=\"$xxljob_cluster\"}, cluster)",
          "refId": "k8s_cluster"
        },
        "refresh": 2,
        "sort": 2,
        "type": "query"
      },
      {
        "current": {
          "text": "https://xxljob.liquiditytech.com/xxl-job-admin/",
          "value": "https://xxljob.liquiditytech.com/xxl-job-admin/"
        },
        "datasource": {
          "type": "prometheus",
          "uid": "${datasource}"
        },
        "definition": "label_values(up{xxljob_cluster=\"$xxljob_cluster\"}, xxljob_dns)",
        "includeAll": false,
        "label": "管理后台",
        "name": "xxljob_dns",
        "options": [],
        "query": {
          "query": "label_values(up{xxljob_cluster=\"$xxljob_cluster\"}, xxljob_dns)",
          "refId": "xxljob_dns"
        },
        "refresh": 2,
        "type": "query"
      },
      {
        "current": {
          "text": "xxl-job",
          "value": "xxl-job"
        },
        "datasource": {
          "type": "prometheus",
          "uid": "${datasource}"
        },
        "definition": "label_values(kube_pod_status_ready{pod=~\"xxl-job.*\"}, namespace)",
        "includeAll": false,
        "label": "命名空间",
        "name": "namespace",
        "options": [],
        "query": {
          "query": "label_values(kube_pod_status_ready{pod=~\"xxl-job.*\"}, namespace)",
          "refId": "namespace"
        },
        "refresh": 2,
        "sort": 1,
        "type": "query"
      }
    ]
  },
  "time": {
    "from": "now-3h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "browser",
  "title": "XXL-Job 监控看板",
  "uid": "xxljob-monitor-v1",
  "version": 26
}

xxljob-dashboard.json

项目价值

复制代码 
维度    价值说明
故障发现    Pod 状态 + 重启次数 + 运行时长三件套，可在 5 分钟刷新周期内捕捉到服务异常/重启事件，缩短 MTTD（平均发现时间）
根因定位    从"整体失败率 → 业务组失败趋势 → 业务组明细 → 24h Job 明细"逐层下钻，从集群级定位到具体 job_name，缩短 MTTI（定位时间）
慢任务定位 ✨ 新增    P95/P99 耗时让"慢任务"问题可量化——以前只能看到"成功了 N 个"，现在能知道"最慢的 1% 花了多久"，支撑性能优化决策（是否拆分任务、是否扩并发、是否优化 SQL）
容量规划    CPU/内存 使用量 vs Limit 叠加曲线，判断是否需要扩容或调限，避免资源浪费与 OOM/Throttling
SLA 度量    1h + 24h 双成功率 Gauge，分别覆盖"实时健康度"和"SLA 达标度"；P99 耗时可作为延迟 SLA 的客观依据
大集群可扩展性 ✨ 优化    24h 明细表加 Top N / 分页后，job_name 再多也不会让看板卡死或信息过载，看板在多团队、多业务线的大集群下仍可用
多集群复用    通过变量模板化，一套看板可覆盖多个 XXL-Job 集群 / K8s 集群，无需重复建板，运维成本低
一键运营闭环    xxljob_dns 变量直通管理后台，发现失败/慢任务后可立即跳转处理，闭环更快
噪音过滤    失败趋势只显示 > 0 的组、明细表默认按失败数降序、24h 表 Top N，三层过滤让运维聚焦"有问题"的部分

XXL-Job 监控看板 --- P0/P1/P2 三级告警方案

复制代码 
# XXL-Job 监控看板 — P0/P1/P2 三级告警方案

> 适用看板：XXL-Job 监控看板（`xxljob-monitor-v1`）
> 覆盖指标：Pod 健康、资源使用、任务成功率/失败率、P95/P99 耗时、业务组/Job 维度明细
> 生成时间：2026-07-30

---

## 一、分级原则

| 级别 | 含义 | 响应时效 | 通知方式 | 触发场景 |
|------|------|---------|---------|---------|
| **P0 Critical** | 服务不可用/严重故障 | 立即（分钟级） | 电话 + 短信 + IM | 全量 Pod DOWN、大面积任务失败、即将 OOM |
| **P1 Warning** | 性能下降/局部异常 | 工作时间内尽快 | IM 群 @值班 | 失败率飙升、P99 异常、资源吃紧 |
| **P2 Info** | 趋势预警/容量告警 | 当日处理 | IM 群通知 | SLA 边缘、渐进性变慢、容量接近上限 |

---

## 二、P0 --- Critical（立即响应）

### 1. Admin Pod DOWN
```promql
max(kube_pod_status_ready{namespace="$namespace", cluster="$k8s_cluster", pod=~"xxl-job-admin.*", condition="true"}) == 0
```
- **阈值**：`== 0`（无 Ready Pod）
- **持续**：`for: 2m`（避免短暂抖动）
- **说明**：Admin 服务完全不可用，调度中枢宕机，所有任务调度停摆

### 2. Pod 频繁重启（CrashLoop）
```promql
sum(max by (namespace, pod, container) (
  increase(kube_pod_container_status_restarts_total{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin"}[1h])
)) >= 5
```
- **阈值**：`≥ 5 次/1h`
- **持续**：`for: 0m`（立即触发）
- **说明**：1 小时内重启 5 次以上，大概率 CrashLoopBackOff，需立即介入

### 3. Ready Pod 数量为 0
```promql
count(max by (namespace, pod) (
  kube_pod_status_ready{namespace="$namespace", cluster="$k8s_cluster", pod=~"xxl-job-admin.*", condition="true"} == 1
)) == 0
```
- **阈值**：`== 0`
- **持续**：`for: 2m`
- **说明**：多副本全挂，与 P0-1 互补，更明确表达"无可用实例"

### 4. 整体任务成功率断崖式下跌（1h）
```promql
sum(xxl_job_task_result_success_count{xxljob_cluster="$xxljob_cluster"})
  / clamp_min(
      sum(xxl_job_task_result_success_count{xxljob_cluster="$xxljob_cluster"})
    + sum(xxl_job_task_result_fail_count{xxljob_cluster="$xxljob_cluster"}),
    1
    ) * 100 < 50
```
- **阈值**：`< 50%`
- **持续**：`for: 5m`
- **说明**：成功率跌破 50% 意味着大面积任务失败，业务影响严重

### 5. 内存即将 OOM
```promql
sum(container_memory_working_set_bytes{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin", image!=""})
  / sum(kube_pod_container_resource_limits{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin", resource="memory"}) > 0.9
```
- **阈值**：`> 90% Limit`
- **持续**：`for: 3m`
- **说明**：内存占用超 90% 上限，距离 OOM Kill 一步之遥

---

## 三、P1 --- Warning（尽快响应）

### 6. 整体任务失败率升高（1h）
```promql
sum(xxl_job_task_result_fail_count{xxljob_cluster="$xxljob_cluster"})
  / clamp_min(
      sum(xxl_job_task_result_success_count{xxljob_cluster="$xxljob_cluster"})
    + sum(xxl_job_task_result_fail_count{xxljob_cluster="$xxljob_cluster"}),
    1
    ) * 100 > 10
```
- **阈值**：`> 10%`
- **持续**：`for: 10m`
- **说明**：失败率超 10%，需排查具体业务组

### 7. P99 执行耗时异常（慢任务）✨
```promql
histogram_quantile(0.99,
  sum(rate(xxl_job_task_duration_seconds_bucket{xxljob_cluster="$xxljob_cluster"}[5m])) by (le)
) > 300
```
- **阈值**：`> 300s`（5 分钟，按业务调整）
- **持续**：`for: 10m`
- **说明**：最慢 1% 任务耗时超阈值，可能阻塞调度线程、影响下游

### 8. 内存高水位
```promql
sum(container_memory_working_set_bytes{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin", image!=""})
  / sum(kube_pod_container_resource_limits{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin", resource="memory"}) > 0.8
```
- **阈值**：`> 80% Limit`
- **持续**：`for: 10m`
- **说明**：内存压力预警，P0 前置防线

### 9. CPU 持续高负载
```promql
sum(rate(container_cpu_usage_seconds_total{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin", image!=""}[5m]))
  / sum(kube_pod_container_resource_limits{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin", resource="cpu"}) > 0.8
```
- **阈值**：`> 80% Limit`
- **持续**：`for: 15m`
- **说明**：CPU 接近 throttle，任务调度可能受影响

### 10. 单业务组失败数突增
```promql
sum by (job_group_name) (
  increase(xxl_job_task_result_fail_count{xxljob_cluster="$xxljob_cluster"}[15m])
) > 10
```
- **阈值**：`> 10 次/15min`（按业务调整）
- **持续**：`for: 5m`
- **说明**：定位到具体业务组，比整体失败率更早发现局部问题

### 11. Pod 出现重启
```promql
sum(max by (namespace, pod, container) (
  increase(kube_pod_container_status_restarts_total{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin"}[1h])
)) >= 1
```
- **阈值**：`≥ 1 次/1h`
- **持续**：`for: 0m`
- **说明**：P0 的前置预警，单次重启也值得看一眼

---

## 四、P2 --- Info（当日处理）

### 12. SLA 边缘（24h 成功率）
```promql
sum(xxl_job_task_daily_success_count{xxljob_cluster="$xxljob_cluster"})
  / clamp_min(
      sum(xxl_job_task_daily_success_count{xxljob_cluster="$xxljob_cluster"})
    + sum(xxl_job_task_daily_fail_count{xxljob_cluster="$xxljob_cluster"}),
    1
    ) * 100 < 95
```
- **阈值**：`< 95%`
- **持续**：`for: 30m`
- **说明**：24h 成功率跌破 SLA 线（95%），需复盘但不紧急

### 13. P95 渐进性变慢（趋势告警）
```promql
histogram_quantile(0.95,
  sum(rate(xxl_job_task_duration_seconds_bucket{xxljob_cluster="$xxljob_cluster"}[1h])) by (le)
) > 120
```
- **阈值**：`> 120s`（按业务调整）
- **持续**：`for: 1h`
- **说明**：长尾任务变慢，"温水煮青蛙"型问题，配合基线对比效果更好

### 14. 容量预警（内存长期高占用）
```promql
avg_over_time(
  sum(container_memory_working_set_bytes{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin", image!=""})
    / sum(kube_pod_container_resource_limits{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin", resource="memory"}) [1h:5m]
) > 0.6
```
- **阈值**：`> 60% 持续 1h`
- **持续**：`for: 1h`
- **说明**：长期占用 60% 以上，提示容量规划，考虑扩容/调 Limit

### 15. 单 Job 出现失败（早期发现）
```promql
sum by (job_group_name, job_name) (
  increase(xxl_job_task_result_fail_count{xxljob_cluster="$xxljob_cluster"}[1h])
) > 0
```
- **阈值**：`> 0`
- **持续**：`for: 15m`
- **说明**：任意 job 出现失败即通知，最细粒度的早期信号

### 16. CPU 长期高占用
```promql
avg_over_time(
  sum(rate(container_cpu_usage_seconds_total{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin", image!=""}[5m]))
    / sum(kube_pod_container_resource_limits{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin", resource="cpu"}) [1h:5m]
) > 0.6
```
- **阈值**：`> 60% 持续 1h`
- **持续**：`for: 1h`
- **说明**：容量预警，同 P2-14

---

## 五、告警配置建议

### 通知路由

| 级别 | 渠道 | 工作时间外 | 升级机制 |
|------|------|-----------|---------|
| P0 | 电话 + 短信 + IM @全体 | 7×24 唤醒 | 5 分钟无 ack → 升级上级 |
| P1 | IM @值班 | 工作时间内 | 30 分钟无 ack → 升级 |
| P2 | IM 群通知 | 次日处理 | 无升级 |

### 抑制与聚合（避免告警风暴）

- **P0-1 Pod DOWN** 触发时，抑制 P1-11 / P2 相关的 Pod 重启告警（同源问题）
- **P0-4 大面积失败** 触发时，抑制 P1-10 / P2-15 的单业务组、单 Job 失败告警
- **P0-5 OOM** 触发时，抑制 P1-8 / P2-14 的内存水位告警
- 同一 `job_group_name` 的告警在 5 分钟内聚合为一条

### 阈值校准建议

- P99/P95 耗时阈值（300s/120s）需根据实际任务画像调整，建议先跑一周观察分布再定基线
- 失败率/失败数阈值需按业务量级调整：小业务 10 次失败是大问题，大业务 100 次可能正常
- 所有阈值上线后建议每季度复盘一次，避免业务规模变化后阈值失效

---

## 六、告警全景速查表

| 编号 | 级别 | 告警名称 | 核心指标 | 阈值 | 持续 |
|------|------|---------|---------|------|------|
| 1 | P0 | Admin Pod DOWN | kube_pod_status_ready | ==0 | 2m |
| 2 | P0 | Pod 频繁重启 | container_status_restarts_total | ≥5/1h | 0m |
| 3 | P0 | Ready Pod 数量为 0 | kube_pod_status_ready count | ==0 | 2m |
| 4 | P0 | 任务成功率断崖下跌 | success/total | <50% | 5m |
| 5 | P0 | 内存即将 OOM | mem/Limit | >90% | 3m |
| 6 | P1 | 任务失败率升高 | fail/total | >10% | 10m |
| 7 | P1 | P99 耗时异常 | histogram_quantile(0.99) | >300s | 10m |
| 8 | P1 | 内存高水位 | mem/Limit | >80% | 10m |
| 9 | P1 | CPU 持续高负载 | cpu/Limit | >80% | 15m |
| 10 | P1 | 单业务组失败突增 | increase(fail) by group | >10/15m | 5m |
| 11 | P1 | Pod 出现重启 | container_status_restarts_total | ≥1/1h | 0m |
| 12 | P2 | SLA 边缘（24h） | daily success/total | <95% | 30m |
| 13 | P2 | P95 渐进变慢 | histogram_quantile(0.95) | >120s | 1h |
| 14 | P2 | 内存长期高占用 | avg_over_time(mem/Limit) | >60% | 1h |
| 15 | P2 | 单 Job 出现失败 | increase(fail) by job | >0 | 15m |
| 16 | P2 | CPU 长期高占用 | avg_over_time(cpu/Limit) | >60% | 1h |

---

*本文档基于 XXL-Job 监控看板（xxljob-monitor-v1）现有指标制定，阈值需结合实际业务画像校准后上线。*

监控指标

待优化说明

复制代码 
告警联动：当前是只读看板，建议接 Grafana Alerting，对"重启次数≥5""失败率>10%""Pod DOWN""P99 > 阈值"配告警规则
耗时基线对比：P95/P99 单看绝对值意义有限，建议叠加"近 7 天同时段均值/基线"，判断当前是否异常飙升
失败原因维度：若指标带 error_type label，可加失败原因分布饼图，从"谁失败了"进一步到"为什么失败"
耗时 vs 失败关联：可做散点图（X=耗时，Y=失败率），找出"又慢又爱失败"的高危任务
相关推荐
做萤石二次开发的哈哈19 小时前
视频解码器怎么对接？解码上墙、电视墙开窗与场景切换的ISAPI接入实战
人工智能·物联网·监控·视频编解码·大屏端·萤石开放平台·蓝海aiot一站式工作台
lisanmengmeng1 天前
nagios 图形化监控部署
监控·nagios
鸿观工坊1 天前
第5篇 Nginx 监控：流量、连接与性能指标
监控
鸿观工坊2 天前
第4篇 Spring Boot 应用监控：Micrometer 与 JVM 指标
监控
鸿观工坊2 天前
第1篇 Prometheus 监控体系全景与基础环境搭建
监控
苏渡苇9 天前
Spring Insight 里如何把 Span 画成瀑布时间线
后端·spring·spring cloud·springboot·监控·apm
行百里er9 天前
轻量级 Spring 监测工具——Spring Insight 发布了
spring boot·后端·监控
宋均浩10 天前
告警规则 243 砍到 27：Prometheus 降噪实战，日均打扰 47 次 → 3 次
云原生·监控·devops
小小ken11 天前
水星摄像头(MIPC552W)双摄版和乔安摄像头(JA-A12,JA-w1)接入easynvr具体步骤
监控·摄像头·nvr·easynvr·水星·乔安
行百里er11 天前
Spring Insight 里如何把 Span 画成瀑布时间线
spring boot·后端·监控
热门推荐
01国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）02历年考研数学一、数学二、数学三真题试卷及答案PDF03Telegram APK 下载安装完整指南 — 2026年最新04GitHub 镜像站点052026 爱知・名古屋亚运会怎么看？直播平台+赛程安排分享06蜘蛛磁力 搜索引擎大全，如何使用蜘蛛磁力查找磁力链接072026 年 9 月国内外主流大模型横向硬核评测：价格、算力、业务场景全维度对比，GPT‑6 Astra横空出世08给AI一张“业务概念地图“：本体如何从哲学走向企业智能09微信历史版本含下载地址（ Windows PC | 安卓 | MAC ）及设置微信不更新10jev-ultrafast 深度解析：7 秒订机票的浏览器 Agent 是如何炼成的