pushgateway指标聚合问题

一 问题现象

一个job有多个实例推送指标,但是从pushgateway上看这个job的instance字段,只显示一个实例的ip,而不是多个实例。导致在grafana上无法正常根据ip查看监控。

应用的prometheus的配置

yaml 复制代码
management:
  metrics:
    tags:
      application: ${spring.application.name}
    export:
      prometheus:
        pushgateway:
          #pushgateway地址
          baseUrl:  ***:9091
          #推送周期
          pushRate: 15s
          #job定义名
          job: ${spring.application.name}
          #启用推送
          enabled: true
  ## actuator暴露出prometheus
  endpoints:
    web:
      exposure:
        include: prometheus
  endpoint:
    health:
      show-details: always

二 解决办法

增加grouping-key,避免多个不同的实例的指标被聚合。HOSTNAME这个环境变量是k8s自带的

yaml 复制代码
management:
  metrics:
    tags:
      application: ${spring.application.name}
    export:
      prometheus:
        pushgateway:
          #pushgateway地址
          baseUrl: ***:9091
          #推送周期
          pushRate: 15s
          #job定义名
          job: ${spring.application.name}
          #启用推送
          enabled: true
          grouping-key:
            hostname: ${HOSTNAME}
  ## actuator暴露出prometheus
  endpoints:
    web:
      exposure:
        include: prometheus,health
  endpoint:
    health:
      show-details: always

三 groupingkey作用分析

pushgateway指标上报的源码路径:io.prometheus.client.exporter.PushGateway#doRequest

没配置groupingkey的时候
配置了groupingkey的时候

由于是本机debug,HostName这个环境变量取不到,k8s上正常。

grouping-key 官方解释 https://github.com/prometheus/pushgateway

The default port the Pushgateway is listening to is 9091. The path looks like

/metrics/job/<JOB_NAME>{/<LABEL_NAME>/<LABEL_VALUE>}

<JOB_NAME> is used as the value of the job label, followed by any number of other label pairs (which might or might not include an instance label). The label set defined by the URL path is used as a grouping key. Any of those labels already set in the body of the request (as regular labels, e.g. name{job="foo"} 42) will be overwritten to match the labels defined by the URL path!

pushgateway是用请求pushgateway的url的path作为grouping-key的。如果不配置任何grouping-key,pushgateway会把所有job一样的指标聚合到一起

相关推荐
bing_1581 天前
如何将 Redis 监控集成到微服务整体的监控体系中( 如 Prometheus + Grafana)
redis·微服务·prometheus
cui_win4 天前
Prometheus实战教程 01 - Prometheus 简介和架构
架构·prometheus
Fireworkitte5 天前
Prometheus + Grafana + Micrometer 监控方案详解
grafana·prometheus
软件测试-阿涛7 天前
【性能测试】Jmeter+Grafana+InfluxDB+Prometheus Windows安装部署教程
测试工具·jmeter·性能优化·压力测试·grafana·prometheus
lpt1111111119 天前
prometheus + grafana 搭建
java·grafana·prometheus
广目软件10 天前
GM DC Monitor v2.1平台部署手册分享
运维·服务器·zabbix·运维开发·prometheus
技术栈壳10 天前
Prometheus+altermanager搭配钉钉报警
prometheus
c_zyer10 天前
Kafka监控体系搭建:基于Prometheus+JMX+Grafana的全方位性能观测方案
kafka·grafana·prometheus·jmx
文人sec11 天前
性能测试-从0到1搭建性能测试环境Jmeter+Grafana+influxDB+Prometheus+Linux
jmeter·grafana·prometheus
风清再凯11 天前
prometheus UI 和node_exporter节点图形化Grafana
ui·grafana·prometheus