pushgateway指标聚合问题

一 问题现象

一个job有多个实例推送指标,但是从pushgateway上看这个job的instance字段,只显示一个实例的ip,而不是多个实例。导致在grafana上无法正常根据ip查看监控。

应用的prometheus的配置

yaml 复制代码
management:
  metrics:
    tags:
      application: ${spring.application.name}
    export:
      prometheus:
        pushgateway:
          #pushgateway地址
          baseUrl:  ***:9091
          #推送周期
          pushRate: 15s
          #job定义名
          job: ${spring.application.name}
          #启用推送
          enabled: true
  ## actuator暴露出prometheus
  endpoints:
    web:
      exposure:
        include: prometheus
  endpoint:
    health:
      show-details: always

二 解决办法

增加grouping-key,避免多个不同的实例的指标被聚合。HOSTNAME这个环境变量是k8s自带的

yaml 复制代码
management:
  metrics:
    tags:
      application: ${spring.application.name}
    export:
      prometheus:
        pushgateway:
          #pushgateway地址
          baseUrl: ***:9091
          #推送周期
          pushRate: 15s
          #job定义名
          job: ${spring.application.name}
          #启用推送
          enabled: true
          grouping-key:
            hostname: ${HOSTNAME}
  ## actuator暴露出prometheus
  endpoints:
    web:
      exposure:
        include: prometheus,health
  endpoint:
    health:
      show-details: always

三 groupingkey作用分析

pushgateway指标上报的源码路径:io.prometheus.client.exporter.PushGateway#doRequest

没配置groupingkey的时候
配置了groupingkey的时候

由于是本机debug,HostName这个环境变量取不到,k8s上正常。

grouping-key 官方解释 https://github.com/prometheus/pushgateway

The default port the Pushgateway is listening to is 9091. The path looks like

/metrics/job/<JOB_NAME>{/<LABEL_NAME>/<LABEL_VALUE>}

<JOB_NAME> is used as the value of the job label, followed by any number of other label pairs (which might or might not include an instance label). The label set defined by the URL path is used as a grouping key. Any of those labels already set in the body of the request (as regular labels, e.g. name{job="foo"} 42) will be overwritten to match the labels defined by the URL path!

pushgateway是用请求pushgateway的url的path作为grouping-key的。如果不配置任何grouping-key,pushgateway会把所有job一样的指标聚合到一起

相关推荐
是垚不是土4 天前
Prometheus接入“飞书“实现自动化告警
运维·安全·自动化·github·飞书·prometheus
朱包林5 天前
Prometheus监控K8S集群-ExternalName-endpoints-ElasticStack采集K8S集群日志实战
运维·云原生·容器·kubernetes·prometheus
3分云计算5 天前
Prometheus 05-01: 告警规则与Alertmanager配置
prometheus
散一世繁华,颠半世琉璃8 天前
Prometheus+Grafana轻松实现SpringBoot应用监控
spring boot·grafana·prometheus
Achou.Wang9 天前
kube-prometheus监控服务发现
服务发现·prometheus
LoneEon10 天前
Ubuntu 系统安装 Prometheus+Grafana(附shell脚本一键部署↓)
ubuntu·grafana·prometheus
文人sec12 天前
性能测试-jmeter15-性能项目计划流
分布式·jmeter·性能优化·grafana·prometheus·模块测试
Knight_AL12 天前
Spring Cloud Gateway 实战:全局过滤器日志统计与 Prometheus + Grafana 接口耗时监控
spring boot·spring cloud·grafana·prometheus
麦兜*13 天前
Redis监控告警体系搭建:使用Redis Exporter + Prometheus + Grafana
java·spring boot·redis·spring·spring cloud·grafana·prometheus