XXL-Job 全链路监控看板和告警方案：从 Pod 健康到任务尾延迟的可观测闭环

meijinmeng2026-07-30 11:10

项目说明

复制代码 
# XXL-Job Exporter 监控

通过 [sql_exporter](https://github.com/burningalchemist/sql_exporter) 直查 XXL-Job RDS 数据库，将任务成功/失败数暴露为 Prometheus 指标，配合 Grafana 看板展示多环境 XXL-Job 运行状态。

## 目录结构

```
xxljob-exporter/
├── grafana-dashboard.json       # Grafana 看板（导入一次，所有环境共用）
├── qa/
│   ├── xxljob-sql-exporter.yaml # QA 环境：ConfigMap + Deployment + Service + ServiceMonitor
│   └── argocd-app.yaml          # QA 环境 ArgoCD Application
├── uat/                         # UAT 环境（参考 qa/ 新建）
├── prod/                        # Prod 环境（参考 qa/ 新建）
└── README.md
```

## 架构与映射原理

```
┌─────────────────────────────────────────────────────────────┐
│  用户在 Grafana 选择: xxljob_cluster = xxljob-qa            │
│                         ↓                                   │
│  隐藏变量自动推导:                                           │
│  $k8s_cluster = label_values(                               │
│    up{xxljob_cluster="xxljob-qa"}, cluster                  │
│  )  →  "ltp-nonprod-eks"                                    │
│                         ↓                                   │
│  sql-exporter 指标: xxljob_cluster="xxljob-qa"              │
│  kube-state-metrics 指标: cluster="ltp-nonprod-eks"         │
└─────────────────────────────────────────────────────────────┘
```

**映射关系不在 dashboard 里维护**，由两处配置自动决定：

| 配置项 | 位置 | 作用 |
|--------|------|------|
| `externalLabels.cluster` | 各集群 Prometheus values.yaml | 决定 `$k8s_cluster` 的值 |
| ServiceMonitor `xxljob_cluster` relabeling | 各环境 `xxljob-sql-exporter.yaml` | 决定 `$xxljob_cluster` 的值 |

**当前环境映射：**

| xxljob_cluster | k8s cluster | 说明 |
|----------------|-------------|------|
| xxljob-qa | ltp-nonprod-eks | QA 环境 |
| xxljob-prod | ltp-eks-prod | 生产环境 |

## 新增环境（以 UAT 为例）

### Step 1 --- 复制 QA 配置

```bash
cp -r xxljob-exporter/qa xxljob-exporter/uat
```

### Step 2 — 修改 `uat/xxljob-sql-exporter.yaml`

需要修改以下字段：

```yaml
# 1. target --- RDS 地址改为 UAT 的
target:
  data_source_name: "mysql://root:${DB_PASSWORD}@<uat-rds-endpoint>:3306/xxl_job?..."

# 2. ServiceMonitor relabelings --- 改为 UAT 的标识
relabelings:
  - targetLabel: xxljob_cluster
    replacement: xxljob-uat          # ← 改这里，全局唯一
  - targetLabel: xxljob_dns
    replacement: https://uat-xxljob.icbc.com/xxl-job-admin/  # ← 改这里
```

### Step 3 — 修改 `uat/argocd-app.yaml`

```yaml
metadata:
  name: xxljob-exporter-uat          # ← 改名

spec:
  source:
    path: xxljob-exporter/uat        # ← 改路径
  destination:
    server: https://<uat-eks-api-endpoint>   # ← UAT 集群的 API server
    namespace: xxl-job
```

### Step 4 --- 创建 DB Secret

在 UAT 集群执行（Secret 不纳入 Git）：

```bash
kubectl create secret generic xxljob-db-secret \
  --from-literal=db-password='<uat-db-password>' \
  -n xxl-job \
  --context <uat-kubeconfig-context>
```

### Step 5 --- 提交并部署

```bash
git add xxljob-exporter/uat/
git commit -m "feat: add xxljob exporter for uat"
git push

# 在 UAT 集群 ArgoCD 中接管
kubectl apply -f xxljob-exporter/uat/argocd-app.yaml -n argocd
```

### Step 6 --- 确认映射生效

部署后约 2 分钟，在 Prometheus 中验证：

```promql
up{xxljob_cluster="xxljob-uat"}
```

返回结果中应同时包含 `cluster="ltp-uat-eks"`（或对应集群名）。

Grafana 看板下拉菜单刷新后自动出现 `xxljob-uat` 选项，**无需修改 dashboard**。

## Grafana 看板导入

1. Grafana → Dashboards → Import
2. 上传 `grafana-dashboard.json`
3. 选择对应的 Prometheus datasource
4. 导入后所有环境共用此看板

看板变量说明：

| 变量 | 类型 | 说明 |
|------|------|------|
| `$datasource` | 数据源 | Prometheus 实例 |
| `$xxljob_cluster` | 下拉选择 | XXL-Job 集群标识，用户可见 |
| `$k8s_cluster` | 隐藏 | 自动从 `up` 指标映射出 K8s 集群名，用于过滤 kube 指标 |
| `$xxljob_dns` | 下拉选择 | 管理后台 URL，随 `$xxljob_cluster` 联动 |
| `$namespace` | 下拉选择 | 随 `$k8s_cluster` 过滤后的命名空间 |

## 指标说明

| 指标名 | 类型 | 说明 | 采集周期 |
|--------|------|------|----------|
| `xxl_job_task_result_success_count` | gauge | 近 1h 各业务组成功数 | 120s |
| `xxl_job_task_result_fail_count` | gauge | 近 1h 各业务组失败数 | 120s |
| `xxl_job_task_result_running_count` | gauge | 近 1h 各业务组运行中数 | 120s |
| `xxl_job_task_daily_success_count` | gauge | 近 24h 各 Job 成功数 | 300s |
| `xxl_job_task_daily_fail_count` | gauge | 近 24h 各 Job 失败数 | 300s |

## 故障排查

**下拉菜单集群/管理后台为空**

```promql
# 验证 up 指标是否携带 xxljob_cluster 标签
up{xxljob_cluster=~".+"}
```

若无结果，检查：
1. sql-exporter pod 是否运行：`kubectl get pods -n xxl-job`
2. Prometheus Targets 页面是否能发现该 exporter
3. ServiceMonitor namespace 与 Prometheus 是否在同一集群

**Pod 数量显示不正确（偏多）**

多个集群的 kube-state-metrics 指标汇聚到同一 Prometheus 时会出现重复计数。
看板已通过 `cluster="$k8s_cluster"` + `max by (namespace, pod)` 处理去重，
若仍有问题确认 `$k8s_cluster` 变量是否正确解析。

**SQL 查询无数据**

确认 RDS Security Group 允许 EKS Node → 3306，并验证 DB 密码 Secret 存在：

```bash
kubectl get secret xxljob-db-secret -n xxl-job
```

View Code

功能说明

复制代码 
1. 服务健康监控
Pod 状态：UP/DOWN 状态指示，一眼判断 Admin 是否存活
重启次数（1h）：阈值告警（0→绿，1→黄，5→红），快速发现 CrashLoop
Pod 运行时长：判断是否近期被重启过
Ready Pod 数量：多副本场景下反映可用实例数
CPU / Memory Gauge：当前资源占用与上限对比，超限风险可视
2. 资源使用趋势
CPU 使用量：实际使用 vs Limit 两条线叠加，资源争抢一目了然
内存使用量：同上，OOM 风险监控
3. 任务总览（1h + 24h 双时间窗口）
整体成功率 Gauge：近 1h 和近 24h 各一个，短期波动 + 长期趋势对比
成功/失败/运行中任务数：核心 KPI 指标卡
总执行数 / 整体失败率：吞吐量与质量双视角
4. 任务执行趋势
成功数（绿）、失败数（红）、运行中 三条曲线，每分钟采样
失败曲线有 hideSeriesFrom 配置，默认隐藏失败数，聚焦成功趋势时仍可切换查看
5. 任务执行耗时分位数 ✨ 新增
P95 / P99 执行耗时：反映长尾任务的执行时长分布
区分"成功任务的耗时分布"与"整体延迟水位"，支撑慢任务排查
与成功/失败计数互补，从"量"（做了多少）扩展到"质"（做得多慢）
6. 业务组详情
各业务组失败趋势：仅展示 > 0 的失败组，过滤噪音
业务组明细表：成功数/失败数/成功率三列，按失败数排序，带渐变背景色，快速定位最差业务组
7. 24h Job 明细 ✨ 优化
粒度细化到 job_name 级别（业务组 + 任务名称）
Top N 限制 / 分页：大集群下 job_name 众多时，只展示最值得关注的部分（如失败数 Top N 或耗时 Top N），避免行数爆炸、表格卡顿
按失败数排序，成功率渐变着色
8. 模板变量（联动筛选）
变量    说明
datasource    数据源选择
xxljob_cluster    XXL-Job 集群（主变量）
k8s_cluster    自动联动 K8s 集群（hide=1，隐藏不可见）
xxljob_dns    管理后台地址，可一键跳转
namespace    命名空间
整体刷新 5 分钟，时间默认近 3 小时

项目实现

复制代码 
# =============================================================================
# XXL-Job SQL Exporter --- QA 环境
# =============================================================================
# 通过直查 RDS xxl_job_log 表，将任务成功/失败数暴露为 Prometheus 指标
#
# 依赖：
#   - xxl-job namespace 已存在（或 ArgoCD syncOptions 开启 CreateNamespace）
#   - RDS Security Group 允许 EKS Node → 3306
#   - Prometheus Operator 已部署（ServiceMonitor CRD 存在）
#
# Secret 说明：
#   DB 密码当前存于 xxljob-db-secret，请勿将真实密码提交 Git。
#   生产环境建议替换为 SealedSecret 或 External Secrets Operator (AWS Secrets Manager)。
#
# 更新 SQL 查询：
#   修改下方 ConfigMap data.config.yml 中的 query 字段，ArgoCD Sync 后自动生效。
# =============================================================================

# Secret xxljob-db-secret 通过 kubectl create secret 手动创建，不纳入 Git 管理。
# kubectl create secret generic xxljob-db-secret --from-literal=db-password='xxx' -n xxl-job

# -----------------------------------------------------------------------------
# ConfigMap --- sql_exporter 查询配置
# -----------------------------------------------------------------------------
apiVersion: v1
kind: ConfigMap
metadata:
  name: xxljob-sql-exporter-config
  namespace: monitoring
data:
  config.yml: |
    global:
      scrape_timeout: 50s
      scrape_timeout_offset: 500ms
      max_connections: 6
      max_idle_connections: 3

    target:
      data_source_name: "mysql://xxl-job:${DB_PASSWORD}@hostname:3306/xxl_job?parseTime=true&loc=UTC"
      collectors: [xxl_job_metrics_1h, xxl_job_metrics_24h, xxl_job_duration_1h]

    collectors:
      # 近 1 小时：按业务组统计，每 60s 采集一次
      - collector_name: xxl_job_metrics_1h
        min_interval: 120s
        metrics:
          - metric_name: xxl_job_task_result_success_count
            type: gauge
            help: "XXL-Job success count by group (last 1h)"
            key_labels: [job_group_name]
            values: [success_count]
            query_ref: task_result_1h

          - metric_name: xxl_job_task_result_fail_count
            type: gauge
            help: "XXL-Job fail count by group (last 1h)"
            key_labels: [job_group_name]
            values: [fail_count]
            query_ref: task_result_1h

          - metric_name: xxl_job_task_result_running_count
            type: gauge
            help: "XXL-Job running count by group (last 1h)"
            key_labels: [job_group_name]
            values: [running_count]
            query_ref: task_result_1h

        queries:
          - query_name: task_result_1h
            query: |
              SELECT
                COALESCE(g.title, CAST(l.job_group AS CHAR)) AS job_group_name,
                COUNT(CASE WHEN l.handle_code = 200 THEN 1 END)                        AS success_count,
                COUNT(CASE WHEN l.handle_code IN (500, 502) THEN 1 END)                AS fail_count,
                COUNT(CASE WHEN l.handle_code = 0 AND l.trigger_code = 200 THEN 1 END) AS running_count
              FROM xxl_job_log l
              LEFT JOIN xxl_job_group g ON l.job_group = g.id
              WHERE l.trigger_time >= DATE_SUB(NOW(), INTERVAL 1 HOUR)
              GROUP BY l.job_group, g.title

      # 近 24 小时：按具体 Job 统计，每 5m 采集一次降低全表扫描压力
      - collector_name: xxl_job_metrics_24h
        min_interval: 300s
        metrics:
          - metric_name: xxl_job_task_daily_success_count
            type: gauge
            help: "XXL-Job success count per job (last 24h)"
            key_labels: [job_group_name, job_name]
            values: [success_count]
            query_ref: task_daily_24h

          - metric_name: xxl_job_task_daily_fail_count
            type: gauge
            help: "XXL-Job fail count per job (last 24h)"
            key_labels: [job_group_name, job_name]
            values: [fail_count]
            query_ref: task_daily_24h

        queries:
          - query_name: task_daily_24h
            query: |
              SELECT
                COALESCE(g.title, CAST(l.job_group AS CHAR)) AS job_group_name,
                COALESCE(i.job_desc, CAST(l.job_id AS CHAR)) AS job_name,
                COUNT(CASE WHEN l.handle_code = 200 THEN 1 END)         AS success_count,
                COUNT(CASE WHEN l.handle_code IN (500, 502) THEN 1 END) AS fail_count
              FROM xxl_job_log l
              LEFT JOIN xxl_job_group g ON l.job_group = g.id
              LEFT JOIN xxl_job_info  i ON l.job_id   = i.id
              WHERE l.trigger_time >= DATE_SUB(NOW(), INTERVAL 24 HOUR)
              GROUP BY l.job_group, g.title, l.job_id, i.job_desc
              ORDER BY fail_count DESC, success_count DESC
              LIMIT 50

      # 执行耗时：avg/max/慢任务数，每 2m 采集，用于排查慢任务
      - collector_name: xxl_job_duration_1h
        min_interval: 120s
        metrics:
          - metric_name: xxl_job_task_avg_duration_seconds
            type: gauge
            help: "XXL-Job avg execution duration per group (last 1h, success only)"
            key_labels: [job_group_name]
            values: [avg_duration_s]
            query_ref: duration_1h

          - metric_name: xxl_job_task_max_duration_seconds
            type: gauge
            help: "XXL-Job max execution duration per group (last 1h, success only)"
            key_labels: [job_group_name]
            values: [max_duration_s]
            query_ref: duration_1h

          - metric_name: xxl_job_task_slow_count
            type: gauge
            help: "XXL-Job tasks exceeding 60s per group (last 1h)"
            key_labels: [job_group_name]
            values: [slow_gt60s]
            query_ref: duration_1h

        queries:
          - query_name: duration_1h
            query: |
              SELECT
                COALESCE(g.title, CAST(l.job_group AS CHAR)) AS job_group_name,
                ROUND(AVG(TIMESTAMPDIFF(SECOND, l.trigger_time, l.handle_time)), 1) AS avg_duration_s,
                MAX(TIMESTAMPDIFF(SECOND, l.trigger_time, l.handle_time))           AS max_duration_s,
                SUM(CASE WHEN TIMESTAMPDIFF(SECOND, l.trigger_time, l.handle_time) > 60 THEN 1 ELSE 0 END) AS slow_gt60s
              FROM xxl_job_log l
              LEFT JOIN xxl_job_group g ON l.job_group = g.id
              WHERE l.trigger_time >= DATE_SUB(NOW(), INTERVAL 1 HOUR)
                AND l.handle_code = 200
                AND l.handle_time IS NOT NULL
                AND l.trigger_time IS NOT NULL
              GROUP BY l.job_group, g.title

---
# -----------------------------------------------------------------------------
# Deployment --- sql_exporter
# -----------------------------------------------------------------------------
apiVersion: apps/v1
kind: Deployment
metadata:
  name: xxljob-sql-exporter
  namespace: monitoring
  labels:
    app: xxljob-sql-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: xxljob-sql-exporter
  template:
    metadata:
      labels:
        app: xxljob-sql-exporter
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9399"
        prometheus.io/path: "/metrics"
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
      containers:
        - name: sql-exporter
          image: burningalchemist/sql_exporter:0.14.1
          imagePullPolicy: IfNotPresent
          args:
            - -config.file=/config/config.yml
          ports:
            - containerPort: 9399
              name: metrics
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: xxljob-db-secret
                  key: db-password
          resources:
            requests:
              cpu: 50m
              memory: 128Mi
            limits:
              cpu: 200m
              memory: 512Mi
          readinessProbe:
            httpGet:
              path: /healthz
              port: 9399
            initialDelaySeconds: 10
            periodSeconds: 30
          livenessProbe:
            httpGet:
              path: /healthz
              port: 9399
            initialDelaySeconds: 30
            periodSeconds: 60
          volumeMounts:
            - name: config
              mountPath: /config
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: xxljob-sql-exporter-config

---
# -----------------------------------------------------------------------------
# Service --- 暴露 metrics 端口
# -----------------------------------------------------------------------------
apiVersion: v1
kind: Service
metadata:
  name: xxljob-sql-exporter
  namespace: monitoring
  labels:
    app: xxljob-sql-exporter
spec:
  selector:
    app: xxljob-sql-exporter
  ports:
    - port: 9399
      targetPort: 9399
      name: metrics

---
# -----------------------------------------------------------------------------
# ServiceMonitor --- Prometheus Operator 自动发现
# -----------------------------------------------------------------------------
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: xxljob-sql-exporter
  namespace: monitoring
  labels:
    release: kube-prom-stack   # ← 改成你的 Prometheus Operator release label
spec:
  selector:
    matchLabels:
      app: xxljob-sql-exporter
  namespaceSelector:
    matchNames:
      - monitoring
  endpoints:
    - port: metrics
      interval: 60s
      scrapeTimeout: 58s
      relabelings:
        - targetLabel: xxljob_cluster
          replacement: xxljob-prod
        - targetLabel: xxljob_dns
          replacement: https://xxljob.icbc.com/xxl-job-admin/


---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: xxljob-exporter-prod
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/icbc/devops-deploy.git
    targetRevision: main
    path: xxljob-exporter/prod
    directory:
      exclude: 'argocd-app.yaml'
  destination:
    server: https://icbc.eks.amazonaws.com
    namespace: monitoring

  # 首次接管步骤：
  #   1. kubectl apply -f argocd-app.yaml -n argocd
  #   2. ArgoCD UI → xxljob-exporter-qa → Refresh → 确认 diff 无误
  #   3. 手动点 Sync 或开启下方 automated
  syncPolicy:
    # automated:
    #   prune: false
    #   selfHeal: false
    retry:
      limit: 3
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
    syncOptions:
      - ApplyOutOfSyncOnly=true
      - ServerSideApply=true
      - CreateNamespace=true

deploy & argocd-app

Dashboard监控看板

复制代码 
{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana",
          "uid": "-- Grafana --"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "description": "XXL-Job Admin 监控看板 --- Pod 健康、资源使用、任务成功/失败率",
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 1,
  "id": 515,
  "links": [],
  "panels": [
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "id": 100,
      "panels": [],
      "title": "服务健康",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [
            {
              "options": {
                "0": {
                  "color": "red",
                  "index": 0,
                  "text": "DOWN"
                },
                "1": {
                  "color": "green",
                  "index": 1,
                  "text": "UP"
                }
              },
              "type": "value"
            }
          ],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "red",
                "value": 0
              },
              {
                "color": "green",
                "value": 1
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 0,
        "y": 1
      },
      "id": 1,
      "options": {
        "colorMode": "background",
        "graphMode": "none",
        "justifyMode": "center",
        "orientation": "auto",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "auto",
        "wideLayout": true
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "max(kube_pod_status_ready{namespace=\"$namespace\", cluster=\"$k8s_cluster\", pod=~\"xxl-job-admin.*\", condition=\"true\"})",
          "legendFormat": "Admin Pod",
          "refId": "A"
        }
      ],
      "title": "Admin Pod 状态",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "yellow",
                "value": 1
              },
              {
                "color": "red",
                "value": 5
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 4,
        "y": 1
      },
      "id": 2,
      "options": {
        "colorMode": "background",
        "graphMode": "none",
        "justifyMode": "center",
        "orientation": "auto",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "auto",
        "wideLayout": true
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(max by (namespace, pod, container) (increase(kube_pod_container_status_restarts_total{namespace=\"$namespace\", cluster=\"$k8s_cluster\", container=\"xxl-job-admin\"}[1h])))",
          "legendFormat": "重启次数(1h)",
          "refId": "A"
        }
      ],
      "title": "Pod 重启次数 (1h)",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "blue",
                "value": 0
              }
            ]
          },
          "unit": "s"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 8,
        "y": 1
      },
      "id": 3,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "center",
        "orientation": "auto",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "auto",
        "wideLayout": true
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "time() - max(kube_pod_start_time{namespace=\"$namespace\", cluster=\"$k8s_cluster\", pod=~\"xxl-job-admin.*\"})",
          "legendFormat": "运行时长",
          "refId": "A"
        }
      ],
      "title": "Pod 运行时长",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "green",
                "value": 1
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 12,
        "y": 1
      },
      "id": 4,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "center",
        "orientation": "auto",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "auto",
        "wideLayout": true
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "count(max by (namespace, pod) (kube_pod_status_ready{namespace=\"$namespace\", cluster=\"$k8s_cluster\", pod=~\"xxl-job-admin.*\", condition=\"true\"}) == 1)",
          "legendFormat": "Ready Pods",
          "refId": "A"
        }
      ],
      "title": "Ready Pod 数量",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "decimals": 3,
          "mappings": [],
          "max": 2,
          "min": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "yellow",
                "value": 0.5
              },
              {
                "color": "red",
                "value": 1
              }
            ]
          },
          "unit": "none"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 5,
        "w": 6,
        "x": 0,
        "y": 5
      },
      "id": 19,
      "options": {
        "minVizHeight": 75,
        "minVizWidth": 75,
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true,
        "sizing": "auto",
        "text": {}
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"$namespace\", cluster=\"$k8s_cluster\", container=\"xxl-job-admin\", image!=\"\"}[5m]))",
          "legendFormat": "CPU (cores)",
          "refId": "A"
        }
      ],
      "title": "CPU 使用量 (cores)",
      "type": "gauge"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "max": 2147483648,
          "min": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "yellow",
                "value": 1073741824
              },
              {
                "color": "red",
                "value": 1610612736
              }
            ]
          },
          "unit": "bytes"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 5,
        "w": 6,
        "x": 6,
        "y": 5
      },
      "id": 20,
      "options": {
        "minVizHeight": 75,
        "minVizWidth": 75,
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true,
        "sizing": "auto",
        "text": {}
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(container_memory_working_set_bytes{namespace=\"$namespace\", cluster=\"$k8s_cluster\", container=\"xxl-job-admin\", image!=\"\"})",
          "legendFormat": "Memory",
          "refId": "A"
        }
      ],
      "title": "Memory 使用量",
      "type": "gauge"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 10
      },
      "id": 101,
      "panels": [],
      "title": "资源使用",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 2,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "none"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 7,
        "w": 12,
        "x": 0,
        "y": 11
      },
      "id": 5,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "max",
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "multi",
          "sort": "none"
        }
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"$namespace\", cluster=\"$k8s_cluster\", container=\"xxl-job-admin\", image!=\"\"}[5m]))",
          "legendFormat": "CPU Usage",
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(kube_pod_container_resource_limits{namespace=\"$namespace\", cluster=\"$k8s_cluster\", container=\"xxl-job-admin\", resource=\"cpu\"})",
          "legendFormat": "CPU Limit",
          "refId": "B"
        }
      ],
      "title": "CPU 使用量 (cores)",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 2,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "bytes"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 7,
        "w": 12,
        "x": 12,
        "y": 11
      },
      "id": 6,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "max",
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "multi",
          "sort": "none"
        }
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(container_memory_working_set_bytes{namespace=\"$namespace\", cluster=\"$k8s_cluster\", container=\"xxl-job-admin\", image!=\"\"})",
          "legendFormat": "Memory Usage",
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(kube_pod_container_resource_limits{namespace=\"$namespace\", cluster=\"$k8s_cluster\", container=\"xxl-job-admin\", resource=\"memory\"})",
          "legendFormat": "Memory Limit",
          "refId": "B"
        }
      ],
      "title": "内存使用量",
      "type": "timeseries"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 18
      },
      "id": 102,
      "panels": [],
      "title": "任务总览 (近 1 小时)",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "max": 100,
          "min": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "red",
                "value": 0
              },
              {
                "color": "yellow",
                "value": 80
              },
              {
                "color": "green",
                "value": 95
              }
            ]
          },
          "unit": "percent"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 6,
        "w": 6,
        "x": 0,
        "y": 19
      },
      "id": 7,
      "options": {
        "minVizHeight": 75,
        "minVizWidth": 75,
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true,
        "sizing": "auto"
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(xxl_job_task_result_success_count{xxljob_cluster=\"$xxljob_cluster\"}) / clamp_min(sum(xxl_job_task_result_success_count{xxljob_cluster=\"$xxljob_cluster\"}) + sum(xxl_job_task_result_fail_count{xxljob_cluster=\"$xxljob_cluster\"}), 1) * 100",
          "legendFormat": "成功率",
          "refId": "A"
        }
      ],
      "title": "整体任务成功率 (近 1h)",
      "type": "gauge"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 4,
        "x": 6,
        "y": 19
      },
      "id": 8,
      "options": {
        "colorMode": "value",
        "graphMode": "area",
        "justifyMode": "center",
        "orientation": "auto",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "auto",
        "wideLayout": true
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(xxl_job_task_result_success_count{xxljob_cluster=\"$xxljob_cluster\"})",
          "legendFormat": "成功数",
          "refId": "A"
        }
      ],
      "title": "成功任务数",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "red",
                "value": 1
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 4,
        "x": 10,
        "y": 19
      },
      "id": 9,
      "options": {
        "colorMode": "background",
        "graphMode": "area",
        "justifyMode": "center",
        "orientation": "auto",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "auto",
        "wideLayout": true
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(xxl_job_task_result_fail_count{xxljob_cluster=\"$xxljob_cluster\"})",
          "legendFormat": "失败数",
          "refId": "A"
        }
      ],
      "title": "失败任务数",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "yellow",
                "value": 10
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 4,
        "x": 14,
        "y": 19
      },
      "id": 10,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "center",
        "orientation": "auto",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "auto",
        "wideLayout": true
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(xxl_job_task_result_running_count{xxljob_cluster=\"$xxljob_cluster\"})",
          "legendFormat": "运行中",
          "refId": "A"
        }
      ],
      "title": "运行中任务数",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "max": 100,
          "min": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "red",
                "value": 0
              },
              {
                "color": "yellow",
                "value": 80
              },
              {
                "color": "green",
                "value": 95
              }
            ]
          },
          "unit": "percent"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 6,
        "w": 6,
        "x": 18,
        "y": 19
      },
      "id": 18,
      "options": {
        "minVizHeight": 75,
        "minVizWidth": 75,
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true,
        "sizing": "auto"
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(xxl_job_task_daily_success_count{xxljob_cluster=\"$xxljob_cluster\"}) / clamp_min(sum(xxl_job_task_daily_success_count{xxljob_cluster=\"$xxljob_cluster\"}) + sum(xxl_job_task_daily_fail_count{xxljob_cluster=\"$xxljob_cluster\"}), 1) * 100",
          "legendFormat": "24h成功率",
          "refId": "A"
        }
      ],
      "title": "整体任务成功率 (近 24h)",
      "type": "gauge"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "blue",
                "value": 0
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 4,
        "x": 6,
        "y": 22
      },
      "id": 11,
      "options": {
        "colorMode": "value",
        "graphMode": "area",
        "justifyMode": "center",
        "orientation": "auto",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "auto",
        "wideLayout": true
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(xxl_job_task_result_success_count{xxljob_cluster=\"$xxljob_cluster\"}) + sum(xxl_job_task_result_fail_count{xxljob_cluster=\"$xxljob_cluster\"})",
          "legendFormat": "总执行数",
          "refId": "A"
        }
      ],
      "title": "总执行数",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "max": 100,
          "min": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "yellow",
                "value": 5
              },
              {
                "color": "red",
                "value": 10
              }
            ]
          },
          "unit": "percent"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 4,
        "x": 10,
        "y": 22
      },
      "id": 12,
      "options": {
        "colorMode": "background",
        "graphMode": "none",
        "justifyMode": "center",
        "orientation": "auto",
        "percentChangeColorMode": "standard",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "auto",
        "wideLayout": true
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(xxl_job_task_result_fail_count{xxljob_cluster=\"$xxljob_cluster\"}) / clamp_min(sum(xxl_job_task_result_success_count{xxljob_cluster=\"$xxljob_cluster\"}) + sum(xxl_job_task_result_fail_count{xxljob_cluster=\"$xxljob_cluster\"}), 1) * 100",
          "legendFormat": "失败率",
          "refId": "A"
        }
      ],
      "title": "整体失败率",
      "type": "stat"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 25
      },
      "id": 103,
      "panels": [],
      "title": "任务趋势",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 15,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 2,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "short"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "失败数"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "red",
                  "mode": "fixed"
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "成功数"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "green",
                  "mode": "fixed"
                }
              }
            ]
          },
          {
            "__systemRef": "hideSeriesFrom",
            "matcher": {
              "id": "byNames",
              "options": {
                "mode": "exclude",
                "names": [
                  "失败数"
                ],
                "prefix": "All except:",
                "readOnly": true
              }
            },
            "properties": [
              {
                "id": "custom.hideFrom",
                "value": {
                  "legend": false,
                  "tooltip": false,
                  "viz": true
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 8,
        "w": 24,
        "x": 0,
        "y": 26
      },
      "id": 13,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "max",
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "right",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "multi",
          "sort": "none"
        }
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(xxl_job_task_result_success_count{xxljob_cluster=\"$xxljob_cluster\"})",
          "interval": "60s",
          "legendFormat": "成功数",
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(xxl_job_task_result_fail_count{xxljob_cluster=\"$xxljob_cluster\"})",
          "interval": "60s",
          "legendFormat": "失败数",
          "refId": "B"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "sum(xxl_job_task_result_running_count{xxljob_cluster=\"$xxljob_cluster\"})",
          "interval": "60s",
          "legendFormat": "运行中",
          "refId": "C"
        }
      ],
      "title": "任务执行趋势 (每分钟采样)",
      "type": "timeseries"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 34
      },
      "id": 104,
      "panels": [],
      "title": "业务组详情",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 2,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 35
      },
      "id": 14,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "max",
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "multi",
          "sort": "none"
        }
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "xxl_job_task_result_fail_count{xxljob_cluster=\"$xxljob_cluster\"} > 0",
          "legendFormat": "{{job_group_name}}",
          "refId": "A"
        }
      ],
      "title": "各业务组失败趋势 (近 1h)",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "custom": {
            "align": "auto",
            "cellOptions": {
              "type": "auto"
            },
            "inspect": false
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "short"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "成功率"
            },
            "properties": [
              {
                "id": "unit",
                "value": "percent"
              },
              {
                "id": "custom.cellOptions",
                "value": {
                  "mode": "gradient",
                  "type": "color-background"
                }
              },
              {
                "id": "thresholds",
                "value": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "red",
                      "value": 0
                    },
                    {
                      "color": "yellow",
                      "value": 80
                    },
                    {
                      "color": "green",
                      "value": 95
                    }
                  ]
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "失败数"
            },
            "properties": [
              {
                "id": "custom.cellOptions",
                "value": {
                  "mode": "gradient",
                  "type": "color-background"
                }
              },
              {
                "id": "thresholds",
                "value": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "green",
                      "value": 0
                    },
                    {
                      "color": "yellow",
                      "value": 1
                    },
                    {
                      "color": "red",
                      "value": 5
                    }
                  ]
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 35
      },
      "id": 15,
      "options": {
        "cellHeight": "sm",
        "footer": {
          "countRows": false,
          "fields": "",
          "reducer": [
            "sum"
          ],
          "show": false
        },
        "showHeader": true,
        "sortBy": [
          {
            "desc": true,
            "displayName": "失败数"
          }
        ]
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "xxl_job_task_result_success_count{xxljob_cluster=\"$xxljob_cluster\"}",
          "format": "table",
          "instant": true,
          "legendFormat": "",
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "xxl_job_task_result_fail_count{xxljob_cluster=\"$xxljob_cluster\"}",
          "format": "table",
          "instant": true,
          "legendFormat": "",
          "refId": "B"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "xxl_job_task_result_success_count{xxljob_cluster=\"$xxljob_cluster\"} / clamp_min(xxl_job_task_result_success_count{xxljob_cluster=\"$xxljob_cluster\"} + xxl_job_task_result_fail_count{xxljob_cluster=\"$xxljob_cluster\"}, 1) * 100",
          "format": "table",
          "instant": true,
          "legendFormat": "",
          "refId": "C"
        }
      ],
      "title": "业务组任务明细 (近 1h, 按失败数排序)",
      "transformations": [
        {
          "id": "merge",
          "options": {}
        },
        {
          "id": "filterFieldsByName",
          "options": {
            "include": {
              "pattern": "^(job_group_name|Value #[ABC])$"
            }
          }
        },
        {
          "id": "organize",
          "options": {
            "renameByName": {
              "Value #A": "成功数",
              "Value #B": "失败数",
              "Value #C": "成功率",
              "job_group_name": "业务组"
            }
          }
        }
      ],
      "type": "table"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 43
      },
      "id": 200,
      "panels": [],
      "title": "执行耗时分析",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "lineWidth": 2,
            "fillOpacity": 5
          },
          "unit": "s"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 44
      },
      "id": 201,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "max",
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "bottom"
        },
        "tooltip": {
          "mode": "multi"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "xxl_job_task_avg_duration_seconds{xxljob_cluster=\"$xxljob_cluster\"}",
          "legendFormat": "{{job_group_name}} avg",
          "refId": "A"
        }
      ],
      "title": "各业务组平均执行耗时 (近 1h)",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "lineWidth": 2,
            "fillOpacity": 5
          },
          "unit": "s"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 44
      },
      "id": 202,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "max",
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "bottom"
        },
        "tooltip": {
          "mode": "multi"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "xxl_job_task_max_duration_seconds{xxljob_cluster=\"$xxljob_cluster\"}",
          "legendFormat": "{{job_group_name}} max",
          "refId": "A"
        }
      ],
      "title": "各业务组最大执行耗时 (近 1h)",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "lineWidth": 2,
            "fillOpacity": 5
          },
          "unit": "percent"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 7,
        "w": 24,
        "x": 0,
        "y": 52
      },
      "id": 203,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "max",
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "bottom"
        },
        "tooltip": {
          "mode": "multi"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "xxl_job_task_slow_count{xxljob_cluster=\"$xxljob_cluster\"} / clamp_min(xxl_job_task_result_success_count{xxljob_cluster=\"$xxljob_cluster\"}, 1) * 100",
          "legendFormat": "{{job_group_name}}",
          "refId": "A"
        }
      ],
      "title": "慢任务率 >60s (近 1h)",
      "type": "timeseries"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 60
      },
      "id": 105,
      "panels": [],
      "title": "24小时 Job 明细",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "custom": {
            "align": "auto",
            "cellOptions": {
              "type": "auto"
            },
            "inspect": false
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": 0
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "short"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "成功率(24h)"
            },
            "properties": [
              {
                "id": "unit",
                "value": "percent"
              },
              {
                "id": "custom.cellOptions",
                "value": {
                  "mode": "gradient",
                  "type": "color-background"
                }
              },
              {
                "id": "thresholds",
                "value": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "red",
                      "value": 0
                    },
                    {
                      "color": "yellow",
                      "value": 80
                    },
                    {
                      "color": "green",
                      "value": 95
                    }
                  ]
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "失败数(24h)"
            },
            "properties": [
              {
                "id": "custom.cellOptions",
                "value": {
                  "mode": "gradient",
                  "type": "color-background"
                }
              },
              {
                "id": "thresholds",
                "value": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "green",
                      "value": 0
                    },
                    {
                      "color": "yellow",
                      "value": 1
                    },
                    {
                      "color": "red",
                      "value": 5
                    }
                  ]
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 10,
        "w": 24,
        "x": 0,
        "y": 61
      },
      "id": 16,
      "options": {
        "cellHeight": "sm",
        "footer": {
          "countRows": false,
          "fields": "",
          "reducer": [
            "sum"
          ],
          "show": false
        },
        "showHeader": true,
        "sortBy": [
          {
            "desc": true,
            "displayName": "失败数(24h)"
          }
        ]
      },
      "pluginVersion": "12.1.10",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "xxl_job_task_daily_success_count{xxljob_cluster=\"$xxljob_cluster\"}",
          "format": "table",
          "instant": true,
          "legendFormat": "",
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "xxl_job_task_daily_fail_count{xxljob_cluster=\"$xxljob_cluster\"}",
          "format": "table",
          "instant": true,
          "legendFormat": "",
          "refId": "B"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "expr": "xxl_job_task_daily_success_count{xxljob_cluster=\"$xxljob_cluster\"} / clamp_min(xxl_job_task_daily_success_count{xxljob_cluster=\"$xxljob_cluster\"} + xxl_job_task_daily_fail_count{xxljob_cluster=\"$xxljob_cluster\"}, 1) * 100",
          "format": "table",
          "instant": true,
          "legendFormat": "",
          "refId": "C"
        }
      ],
      "title": "24小时 Job 执行明细 Top 50 (按失败数排序)",
      "transformations": [
        {
          "id": "merge",
          "options": {}
        },
        {
          "id": "filterFieldsByName",
          "options": {
            "include": {
              "pattern": "^(job_group_name|job_name|Value #[ABC])$"
            }
          }
        },
        {
          "id": "organize",
          "options": {
            "renameByName": {
              "Value #A": "成功数(24h)",
              "Value #B": "失败数(24h)",
              "Value #C": "成功率(24h)",
              "job_group_name": "业务组",
              "job_name": "任务名称"
            }
          }
        }
      ],
      "type": "table"
    }
  ],
  "preload": false,
  "refresh": "5m",
  "schemaVersion": 41,
  "tags": [
    "xxl-job",
    "task-scheduler",
    "monitoring"
  ],
  "templating": {
    "list": [
      {
        "current": {
          "text": "Metrics",
          "value": "fetdvd0e806bkc"
        },
        "includeAll": false,
        "label": "数据源",
        "name": "datasource",
        "options": [],
        "query": "prometheus",
        "refresh": 1,
        "type": "datasource"
      },
      {
        "current": {
          "text": "xxljob-prod",
          "value": "xxljob-prod"
        },
        "datasource": {
          "type": "prometheus",
          "uid": "${datasource}"
        },
        "definition": "label_values(up{xxljob_cluster=~\".+\"}, xxljob_cluster)",
        "includeAll": false,
        "label": "集群",
        "name": "xxljob_cluster",
        "options": [],
        "query": {
          "query": "label_values(up{xxljob_cluster=~\".+\"}, xxljob_cluster)",
          "refId": "xxljob_cluster"
        },
        "refresh": 2,
        "sort": 1,
        "type": "query"
      },
      {
        "current": {
          "text": "ltp-eks-prod",
          "value": "ltp-eks-prod"
        },
        "datasource": {
          "type": "prometheus",
          "uid": "${datasource}"
        },
        "definition": "label_values(up{xxljob_cluster=\"$xxljob_cluster\"}, cluster)",
        "hide": 1,
        "includeAll": false,
        "label": "K8s集群(自动)",
        "name": "k8s_cluster",
        "options": [],
        "query": {
          "query": "label_values(up{xxljob_cluster=\"$xxljob_cluster\"}, cluster)",
          "refId": "k8s_cluster"
        },
        "refresh": 2,
        "sort": 2,
        "type": "query"
      },
      {
        "current": {
          "text": "https://xxljob.liquiditytech.com/xxl-job-admin/",
          "value": "https://xxljob.liquiditytech.com/xxl-job-admin/"
        },
        "datasource": {
          "type": "prometheus",
          "uid": "${datasource}"
        },
        "definition": "label_values(up{xxljob_cluster=\"$xxljob_cluster\"}, xxljob_dns)",
        "includeAll": false,
        "label": "管理后台",
        "name": "xxljob_dns",
        "options": [],
        "query": {
          "query": "label_values(up{xxljob_cluster=\"$xxljob_cluster\"}, xxljob_dns)",
          "refId": "xxljob_dns"
        },
        "refresh": 2,
        "type": "query"
      },
      {
        "current": {
          "text": "xxl-job",
          "value": "xxl-job"
        },
        "datasource": {
          "type": "prometheus",
          "uid": "${datasource}"
        },
        "definition": "label_values(kube_pod_status_ready{pod=~\"xxl-job.*\"}, namespace)",
        "includeAll": false,
        "label": "命名空间",
        "name": "namespace",
        "options": [],
        "query": {
          "query": "label_values(kube_pod_status_ready{pod=~\"xxl-job.*\"}, namespace)",
          "refId": "namespace"
        },
        "refresh": 2,
        "sort": 1,
        "type": "query"
      }
    ]
  },
  "time": {
    "from": "now-3h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "browser",
  "title": "XXL-Job 监控看板",
  "uid": "xxljob-monitor-v1",
  "version": 26
}

xxljob-dashboard.json

项目价值

复制代码 
维度    价值说明
故障发现    Pod 状态 + 重启次数 + 运行时长三件套，可在 5 分钟刷新周期内捕捉到服务异常/重启事件，缩短 MTTD（平均发现时间）
根因定位    从"整体失败率 → 业务组失败趋势 → 业务组明细 → 24h Job 明细"逐层下钻，从集群级定位到具体 job_name，缩短 MTTI（定位时间）
慢任务定位 ✨ 新增    P95/P99 耗时让"慢任务"问题可量化——以前只能看到"成功了 N 个"，现在能知道"最慢的 1% 花了多久"，支撑性能优化决策（是否拆分任务、是否扩并发、是否优化 SQL）
容量规划    CPU/内存 使用量 vs Limit 叠加曲线，判断是否需要扩容或调限，避免资源浪费与 OOM/Throttling
SLA 度量    1h + 24h 双成功率 Gauge，分别覆盖"实时健康度"和"SLA 达标度"；P99 耗时可作为延迟 SLA 的客观依据
大集群可扩展性 ✨ 优化    24h 明细表加 Top N / 分页后，job_name 再多也不会让看板卡死或信息过载，看板在多团队、多业务线的大集群下仍可用
多集群复用    通过变量模板化，一套看板可覆盖多个 XXL-Job 集群 / K8s 集群，无需重复建板，运维成本低
一键运营闭环    xxljob_dns 变量直通管理后台，发现失败/慢任务后可立即跳转处理，闭环更快
噪音过滤    失败趋势只显示 > 0 的组、明细表默认按失败数降序、24h 表 Top N，三层过滤让运维聚焦"有问题"的部分

XXL-Job 监控看板 --- P0/P1/P2 三级告警方案

复制代码 
# XXL-Job 监控看板 — P0/P1/P2 三级告警方案

> 适用看板：XXL-Job 监控看板（`xxljob-monitor-v1`）
> 覆盖指标：Pod 健康、资源使用、任务成功率/失败率、P95/P99 耗时、业务组/Job 维度明细
> 生成时间：2026-07-30

---

## 一、分级原则

| 级别 | 含义 | 响应时效 | 通知方式 | 触发场景 |
|------|------|---------|---------|---------|
| **P0 Critical** | 服务不可用/严重故障 | 立即（分钟级） | 电话 + 短信 + IM | 全量 Pod DOWN、大面积任务失败、即将 OOM |
| **P1 Warning** | 性能下降/局部异常 | 工作时间内尽快 | IM 群 @值班 | 失败率飙升、P99 异常、资源吃紧 |
| **P2 Info** | 趋势预警/容量告警 | 当日处理 | IM 群通知 | SLA 边缘、渐进性变慢、容量接近上限 |

---

## 二、P0 --- Critical（立即响应）

### 1. Admin Pod DOWN
```promql
max(kube_pod_status_ready{namespace="$namespace", cluster="$k8s_cluster", pod=~"xxl-job-admin.*", condition="true"}) == 0
```
- **阈值**：`== 0`（无 Ready Pod）
- **持续**：`for: 2m`（避免短暂抖动）
- **说明**：Admin 服务完全不可用，调度中枢宕机，所有任务调度停摆

### 2. Pod 频繁重启（CrashLoop）
```promql
sum(max by (namespace, pod, container) (
  increase(kube_pod_container_status_restarts_total{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin"}[1h])
)) >= 5
```
- **阈值**：`≥ 5 次/1h`
- **持续**：`for: 0m`（立即触发）
- **说明**：1 小时内重启 5 次以上，大概率 CrashLoopBackOff，需立即介入

### 3. Ready Pod 数量为 0
```promql
count(max by (namespace, pod) (
  kube_pod_status_ready{namespace="$namespace", cluster="$k8s_cluster", pod=~"xxl-job-admin.*", condition="true"} == 1
)) == 0
```
- **阈值**：`== 0`
- **持续**：`for: 2m`
- **说明**：多副本全挂，与 P0-1 互补，更明确表达"无可用实例"

### 4. 整体任务成功率断崖式下跌（1h）
```promql
sum(xxl_job_task_result_success_count{xxljob_cluster="$xxljob_cluster"})
  / clamp_min(
      sum(xxl_job_task_result_success_count{xxljob_cluster="$xxljob_cluster"})
    + sum(xxl_job_task_result_fail_count{xxljob_cluster="$xxljob_cluster"}),
    1
    ) * 100 < 50
```
- **阈值**：`< 50%`
- **持续**：`for: 5m`
- **说明**：成功率跌破 50% 意味着大面积任务失败，业务影响严重

### 5. 内存即将 OOM
```promql
sum(container_memory_working_set_bytes{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin", image!=""})
  / sum(kube_pod_container_resource_limits{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin", resource="memory"}) > 0.9
```
- **阈值**：`> 90% Limit`
- **持续**：`for: 3m`
- **说明**：内存占用超 90% 上限，距离 OOM Kill 一步之遥

---

## 三、P1 --- Warning（尽快响应）

### 6. 整体任务失败率升高（1h）
```promql
sum(xxl_job_task_result_fail_count{xxljob_cluster="$xxljob_cluster"})
  / clamp_min(
      sum(xxl_job_task_result_success_count{xxljob_cluster="$xxljob_cluster"})
    + sum(xxl_job_task_result_fail_count{xxljob_cluster="$xxljob_cluster"}),
    1
    ) * 100 > 10
```
- **阈值**：`> 10%`
- **持续**：`for: 10m`
- **说明**：失败率超 10%，需排查具体业务组

### 7. P99 执行耗时异常（慢任务）✨
```promql
histogram_quantile(0.99,
  sum(rate(xxl_job_task_duration_seconds_bucket{xxljob_cluster="$xxljob_cluster"}[5m])) by (le)
) > 300
```
- **阈值**：`> 300s`（5 分钟，按业务调整）
- **持续**：`for: 10m`
- **说明**：最慢 1% 任务耗时超阈值，可能阻塞调度线程、影响下游

### 8. 内存高水位
```promql
sum(container_memory_working_set_bytes{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin", image!=""})
  / sum(kube_pod_container_resource_limits{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin", resource="memory"}) > 0.8
```
- **阈值**：`> 80% Limit`
- **持续**：`for: 10m`
- **说明**：内存压力预警，P0 前置防线

### 9. CPU 持续高负载
```promql
sum(rate(container_cpu_usage_seconds_total{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin", image!=""}[5m]))
  / sum(kube_pod_container_resource_limits{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin", resource="cpu"}) > 0.8
```
- **阈值**：`> 80% Limit`
- **持续**：`for: 15m`
- **说明**：CPU 接近 throttle，任务调度可能受影响

### 10. 单业务组失败数突增
```promql
sum by (job_group_name) (
  increase(xxl_job_task_result_fail_count{xxljob_cluster="$xxljob_cluster"}[15m])
) > 10
```
- **阈值**：`> 10 次/15min`（按业务调整）
- **持续**：`for: 5m`
- **说明**：定位到具体业务组，比整体失败率更早发现局部问题

### 11. Pod 出现重启
```promql
sum(max by (namespace, pod, container) (
  increase(kube_pod_container_status_restarts_total{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin"}[1h])
)) >= 1
```
- **阈值**：`≥ 1 次/1h`
- **持续**：`for: 0m`
- **说明**：P0 的前置预警，单次重启也值得看一眼

---

## 四、P2 --- Info（当日处理）

### 12. SLA 边缘（24h 成功率）
```promql
sum(xxl_job_task_daily_success_count{xxljob_cluster="$xxljob_cluster"})
  / clamp_min(
      sum(xxl_job_task_daily_success_count{xxljob_cluster="$xxljob_cluster"})
    + sum(xxl_job_task_daily_fail_count{xxljob_cluster="$xxljob_cluster"}),
    1
    ) * 100 < 95
```
- **阈值**：`< 95%`
- **持续**：`for: 30m`
- **说明**：24h 成功率跌破 SLA 线（95%），需复盘但不紧急

### 13. P95 渐进性变慢（趋势告警）
```promql
histogram_quantile(0.95,
  sum(rate(xxl_job_task_duration_seconds_bucket{xxljob_cluster="$xxljob_cluster"}[1h])) by (le)
) > 120
```
- **阈值**：`> 120s`（按业务调整）
- **持续**：`for: 1h`
- **说明**：长尾任务变慢，"温水煮青蛙"型问题，配合基线对比效果更好

### 14. 容量预警（内存长期高占用）
```promql
avg_over_time(
  sum(container_memory_working_set_bytes{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin", image!=""})
    / sum(kube_pod_container_resource_limits{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin", resource="memory"}) [1h:5m]
) > 0.6
```
- **阈值**：`> 60% 持续 1h`
- **持续**：`for: 1h`
- **说明**：长期占用 60% 以上，提示容量规划，考虑扩容/调 Limit

### 15. 单 Job 出现失败（早期发现）
```promql
sum by (job_group_name, job_name) (
  increase(xxl_job_task_result_fail_count{xxljob_cluster="$xxljob_cluster"}[1h])
) > 0
```
- **阈值**：`> 0`
- **持续**：`for: 15m`
- **说明**：任意 job 出现失败即通知，最细粒度的早期信号

### 16. CPU 长期高占用
```promql
avg_over_time(
  sum(rate(container_cpu_usage_seconds_total{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin", image!=""}[5m]))
    / sum(kube_pod_container_resource_limits{namespace="$namespace", cluster="$k8s_cluster", container="xxl-job-admin", resource="cpu"}) [1h:5m]
) > 0.6
```
- **阈值**：`> 60% 持续 1h`
- **持续**：`for: 1h`
- **说明**：容量预警，同 P2-14

---

## 五、告警配置建议

### 通知路由

| 级别 | 渠道 | 工作时间外 | 升级机制 |
|------|------|-----------|---------|
| P0 | 电话 + 短信 + IM @全体 | 7×24 唤醒 | 5 分钟无 ack → 升级上级 |
| P1 | IM @值班 | 工作时间内 | 30 分钟无 ack → 升级 |
| P2 | IM 群通知 | 次日处理 | 无升级 |

### 抑制与聚合（避免告警风暴）

- **P0-1 Pod DOWN** 触发时，抑制 P1-11 / P2 相关的 Pod 重启告警（同源问题）
- **P0-4 大面积失败** 触发时，抑制 P1-10 / P2-15 的单业务组、单 Job 失败告警
- **P0-5 OOM** 触发时，抑制 P1-8 / P2-14 的内存水位告警
- 同一 `job_group_name` 的告警在 5 分钟内聚合为一条

### 阈值校准建议

- P99/P95 耗时阈值（300s/120s）需根据实际任务画像调整，建议先跑一周观察分布再定基线
- 失败率/失败数阈值需按业务量级调整：小业务 10 次失败是大问题，大业务 100 次可能正常
- 所有阈值上线后建议每季度复盘一次，避免业务规模变化后阈值失效

---

## 六、告警全景速查表

| 编号 | 级别 | 告警名称 | 核心指标 | 阈值 | 持续 |
|------|------|---------|---------|------|------|
| 1 | P0 | Admin Pod DOWN | kube_pod_status_ready | ==0 | 2m |
| 2 | P0 | Pod 频繁重启 | container_status_restarts_total | ≥5/1h | 0m |
| 3 | P0 | Ready Pod 数量为 0 | kube_pod_status_ready count | ==0 | 2m |
| 4 | P0 | 任务成功率断崖下跌 | success/total | <50% | 5m |
| 5 | P0 | 内存即将 OOM | mem/Limit | >90% | 3m |
| 6 | P1 | 任务失败率升高 | fail/total | >10% | 10m |
| 7 | P1 | P99 耗时异常 | histogram_quantile(0.99) | >300s | 10m |
| 8 | P1 | 内存高水位 | mem/Limit | >80% | 10m |
| 9 | P1 | CPU 持续高负载 | cpu/Limit | >80% | 15m |
| 10 | P1 | 单业务组失败突增 | increase(fail) by group | >10/15m | 5m |
| 11 | P1 | Pod 出现重启 | container_status_restarts_total | ≥1/1h | 0m |
| 12 | P2 | SLA 边缘（24h） | daily success/total | <95% | 30m |
| 13 | P2 | P95 渐进变慢 | histogram_quantile(0.95) | >120s | 1h |
| 14 | P2 | 内存长期高占用 | avg_over_time(mem/Limit) | >60% | 1h |
| 15 | P2 | 单 Job 出现失败 | increase(fail) by job | >0 | 15m |
| 16 | P2 | CPU 长期高占用 | avg_over_time(cpu/Limit) | >60% | 1h |

---

*本文档基于 XXL-Job 监控看板（xxljob-monitor-v1）现有指标制定，阈值需结合实际业务画像校准后上线。*

监控指标

待优化说明

复制代码 
告警联动：当前是只读看板，建议接 Grafana Alerting，对"重启次数≥5""失败率>10%""Pod DOWN""P99 > 阈值"配告警规则
耗时基线对比：P95/P99 单看绝对值意义有限，建议叠加"近 7 天同时段均值/基线"，判断当前是否异常飙升
失败原因维度：若指标带 error_type label，可加失败原因分布饼图，从"谁失败了"进一步到"为什么失败"
耗时 vs 失败关联：可做散点图（X=耗时，Y=失败率），找出"又慢又爱失败"的高危任务
相关推荐
Patrick_Wilson15 天前
修好 bug 只是开始：一次由监控需求反向重构日志结构的复盘
前端·监控·数据可视化
阿里云云原生17 天前
终结前端“黑盒”难题：通过 Session Replay 与三维热力图实现像素级用户行为洞察
云原生·监控
带娃的IT创业者17 天前
监控并非安全：当隐私成为技术的祭品
java·开发语言·安全·数据安全·监控·隐私保护·加密技术
翼龙云_cloud23 天前
阿里云国际代理商：阿里云 ECS 全维度监控配置教程
运维·阿里云·云计算·监控
SkyWalking中文站24 天前
认识 Horizon UI · AI Assistant：用日常语言查询你的可观测性数据
运维·监控·自动化运维
SkyWalking中文站24 天前
认识 Horizon UI · 17/17：安装与迁移
运维·监控·自动化运维
SkyWalking中文站24 天前
认识 Horizon UI · 16/17：八种语言的本地化
运维·监控·自动化运维
七夜zippoe24 天前
DolphinDB生产指标监控：KPI实时计算
监控·生产指标·kpi·dolphindb·实时 计算
SkyWalking中文站24 天前
认识 Horizon UI · 15/17：用模板定制控制台
运维·监控·自动化运维
热门推荐
01如何新建文件夹？ 电脑新建文件夹的4种方法02GitHub 镜像站点032026年7月AI圈大地震：GPT-5.6被政府限制、Claude入驻Slack、Anthropic自研芯片04国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）05AI科技热点日报 | 2026年07月01日06AI 编程 IDE 全景解析 2026：Agent 全面接管开发链路072026 国产 AI 大模型横评：DeepSeek、通义千问、Kimi、文心一言、星火、豆包谁更能打？08微信历史版本含下载地址（ Windows PC | 安卓 | MAC ）及设置微信不更新09Agnes AI 免费 API 接入指南：文本、生图、生视频，一套接口全免费102026 年 AI 大模型 & AI 编程工具实战全总结