【Prometheus·部署篇】配置详解：scrape_config、relabeling 与服务发现

前言

上一篇部署好了 Prometheus，但只有一个 job_name 是远远不够的。生产环境需要监控几十上百个目标，手动配置 static_configs 不现实。本篇深入 scrape_config 配置，重点讲解 relabeling 和服务发现------Prometheus 配置的核心能力。

一、scrape_config 完整结构

yaml 复制代码

scrape_configs:
  - job_name: 'example'              # 任务名，会成为标签
    scrape_interval: 15s              # 采集间隔（覆盖全局）
    scrape_timeout: 10s               # 超时时间
    metrics_path: /metrics            # 指标路径
    scheme: http                      # http 或 https
    honor_labels: false               # 是否保留目标自身标签
    honor_timestamps: true            # 是否保留目标时间戳
    params:                           # URL 参数
      module: [http_2xx]

    # 服务发现配置（三选一）
    static_configs:                   # 静态目标
      - targets: ['host1:9100']
        labels:
          env: production

    kubernetes_sd_configs:             # K8s 服务发现
      - role: pod

    consul_sd_configs:                 # Consul 服务发现
      - server: 'consul:8500'

    # Relabel 配置（采集前/后修改标签）
    relabel_configs:                   # 采集前重写标签
      - ...                           # 修改 __address__ 等

    metric_relabel_configs:            # 采集后重写指标
      - ...                           # 修改指标值/标签

    # 认证配置
    basic_auth:                       # Basic 认证
      username: admin
      password: secret

    tls_config:                       # TLS 配置
      ca_file: /path/ca.crt
      cert_file: /path/client.crt
      key_file: /path/client.key
      insecure_skip_verify: false

    bearer_token: "token123"          # Bearer Token

    # 代理
    proxy_url: http://proxy:3128

二、static_configs 静态目标

最简单的方式，手动列出目标：

yaml 复制代码

scrape_configs:
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node1:9100', 'node2:9100', 'node3:9100']
        labels:
          env: 'production'
          region: 'beijing'
      - targets: ['node4:9100', 'node5:9100']
        labels:
          env: 'staging'
          region: 'shanghai'

适用场景：目标数量少、固定不变的环境。

三、Relabeling 详解

Relabeling 是 Prometheus 配置中最强大的功能之一，允许在采集前后动态修改标签。

两个阶段

复制代码

目标发现 → relabel_configs → 采集 /metrics → metric_relabel_configs → 存储
           (修改目标标签)                    (修改指标标签)

基本语法

yaml 复制代码

relabel_configs:
  - source_labels: [__meta_kubernetes_pod_name]  # 源标签
    action: replace                               # 操作类型
    target_label: pod                             # 目标标签
    regex: (.+)                                   # 正则匹配
    replacement: $1                               # 替换值
    separator: ;                                  # 多标签分隔符

Action 类型

Action	说明
replace	用正则替换值（默认）
keep	保留匹配的目标，丢弃不匹配的
drop	丢弃匹配的目标
labelmap	批量重映射标签
labeldrop	删除标签
labelkeep	保留标签（删除其余）
hashmod	hash 取模（分片）

实战示例

1. 过滤目标（keep/drop）

yaml 复制代码

# 只采集有 prometheus.io/scrape=true 注解的 Pod
relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true

# 排除 kube-system 命名空间
relabel_configs:
  - source_labels: [__meta_kubernetes_namespace]
    action: drop
    regex: kube-system

2. 修改采集地址

yaml 复制代码

# K8s Pod 的 __address__ 是 PodIP:Port
# 需要改为 PodIP:metrics_port
relabel_configs:
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    target_label: __address__
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2

3. 修改 metrics_path

yaml 复制代码

# 根据注解决定路径
relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)

4. 修改 scheme

yaml 复制代码

# 有些应用需要 HTTPS
relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
    action: replace
    target_label: __scheme__
    regex: (https?)

5. 提取标签

yaml 复制代码

# 从 K8s 元数据提取信息作为标签
relabel_configs:
  - source_labels: [__meta_kubernetes_namespace]
    target_label: namespace
  - source_labels: [__meta_kubernetes_pod_name]
    target_label: pod
  - source_labels: [__meta_kubernetes_pod_label_app]
    target_label: app
  - source_labels: [__meta_kubernetes_pod_node_name]
    target_label: node

6. labelmap 批量映射

yaml 复制代码

# 将所有 __meta_kubernetes_pod_label_* 标签去掉前缀
relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
    # __meta_kubernetes_pod_label_app → app
    # __meta_kubernetes_pod_label_version → version

7. labeldrop 删除标签

yaml 复制代码

metric_relabel_configs:
  # 删除高基数标签，防止 TSDB 膨胀
  - action: labeldrop
    regex: 'request_id|trace_id|user_id'

8. hashmod 分片

yaml 复制代码

# 当目标过多时，用多个 Prometheus 分担采集
relabel_configs:
  - source_labels: [__address__]
    action: hashmod
    target_label: __tmp_hash
    modulus: 4     # 模 4
  - source_labels: [__tmp_hash]
    action: keep
    regex: 0        # 只采集 hash % 4 == 0 的目标

四、服务发现（Service Discovery）

1. Kubernetes 服务发现

yaml 复制代码

scrape_configs:
  # 采集所有 Pod（带注解的）
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

  # 采集所有 Service
  - job_name: 'kubernetes-services'
    kubernetes_sd_configs:
      - role: service
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true

  # 采集所有 Node
  - job_name: 'kubernetes-nodes'
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - source_labels: [__address__]
        action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?
        replacement: $1:10250
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecure_skip_verify: true
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

  # 采集 K8s 组件（kube-apiserver）
  - job_name: 'kube-apiserver'
    kubernetes_sd_configs:
      - role: endpoints
        namespaces:
          - name: default
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecure_skip_verify: true
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_name]
        action: keep
        regex: kubernetes

2. Consul 服务发现

yaml 复制代码

scrape_configs:
  - job_name: 'consul-services'
    consul_sd_configs:
      - server: 'consul.example.com:8500'
        token: 'consul-acl-token'
        datacenter: 'dc1'
        services: ['web', 'api']    # 只采集注册了 web/api 的服务
        tags: ['production']          # 按标签过滤
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        action: keep
        regex: .*,production,.*
      - source_labels: [__meta_consul_service]
        target_label: service
      - source_labels: [__meta_consul_node]
        target_label: node

3. DNS 服务发现

yaml 复制代码

scrape_configs:
  - job_name: 'dns-discovery'
    dns_sd_configs:
      - names:
          - 'node-exporter.monitoring.svc.cluster.local'
          - 'mysql-exporter.monitoring.svc.cluster.local'
        type: A
        refresh_interval: 30s

4. EC2 服务发现

yaml 复制代码

scrape_configs:
  - job_name: 'ec2-instances'
    ec2_sd_configs:
      - region: us-east-1
        access_key: AKIA...
        secret_key: ...
        port: 9100
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance_name
      - source_labels: [__meta_ec2_tag_Environment]
        target_label: env
      - source_labels: [__meta_ec2_private_ip]
        target_label: private_ip

5. 文件服务发现

yaml 复制代码

scrape_configs:
  - job_name: 'file-discovery'
    file_sd_configs:
      - files:
          - '/etc/prometheus/targets/*.yml'
          - '/etc/prometheus/targets/*.json'
        refresh_interval: 30s

yaml 复制代码

# /etc/prometheus/targets/servers.yml
- targets:
    - 'web1:9100'
    - 'web2:9100'
  labels:
    env: production
    role: web
- targets:
    - 'db1:9104'
  labels:
    env: production
    role: database

6. HTTP 服务发现

yaml 复制代码

scrape_configs:
  - job_name: 'http-discovery'
    http_sd_configs:
      - url: 'http://config-server.example.com/targets'
        refresh_interval: 60s

json 复制代码

// HTTP 响应格式
[
  {
    "targets": ["web1:9100", "web2:9100"],
    "labels": {
      "env": "production",
      "role": "web"
    }
  }
]

五、metric_relabel_configs 采集后重写

yaml 复制代码

scrape_configs:
  - job_name: 'mysql'
    static_configs:
      - targets: ['mysql:9104']
    metric_relabel_configs:
      # 删除不需要的指标
      - source_labels: [__name__]
        regex: 'mysql_global_status_.*'
        action: drop

      # 重命名指标
      - source_labels: [__name__]
        regex: 'mysql_global_status_(.+)'
        target_label: __name__
        replacement: 'mysql_$1'

      # 删除高基数标签
      - action: labeldrop
        regex: 'connection_id|thread_id'

      # 修改标签值
      - source_labels: [table_schema]
        regex: 'information_schema'
        target_label: table_schema
        replacement: 'system'

六、生产配置模板

yaml 复制代码

# 完整的生产级 scrape_configs
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'production'
    env: 'prod'

scrape_configs:
  # 1. Prometheus 自身
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # 2. K8s Pod 自动发现
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

  # 3. Node Exporter DaemonSet
  - job_name: 'node-exporter'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: keep
        regex: node-exporter

  # 4. 外部服务器
  - job_name: 'external-servers'
    file_sd_configs:
      - files: ['/etc/prometheus/targets/external/*.yml']
    metric_relabel_configs:
      - action: labeldrop
        regex: 'request_id|trace_id'

  # 5. Consul 注册的服务
  - job_name: 'consul-services'
    consul_sd_configs:
      - server: 'consul:8500'
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        action: keep
        regex: .*,production,.*

⚠️ 踩坑提示 ：

relabel_configs 在采集前修改，metric_relabel_configs 在采集后修改

__ 开头的标签是内部标签，不会存储到 TSDB

用 labeldrop 清理高基数标签，防止 TSDB 膨胀

file_sd_configs 修改文件后自动生效，不需要 reload

要点回顾

配置项	作用
static_configs	手动指定目标
kubernetes_sd_configs	K8s 服务发现
consul_sd_configs	Consul 服务发现
file_sd_configs	文件服务发现
relabel_configs	采集前修改标签
metric_relabel_configs	采集后修改指标

Relabeling 两大阶段：采集前（修改目标）、采集后（修改指标）
action：replace/keep/drop/labelmap/labeldrop/hashmod
K8s 服务发现 4 种 role：node/pod/service/endpoints
标签以 __ 开头的是内部标签，不入库
生产环境用 file_sd + http_sd + k8s_sd 组合

下一篇预告

采集和存储搞定了，下一篇 【Prometheus·部署篇】存储与容量规划：本地存储、远程存储与长期保存 将讲解 TSDB 存储机制和长期存储方案。