kafka监控配置和告警配置——筑梦之路

kafka_exporter项目地址:https://github.com/danielqsj/kafka_exporter

docker-compose部署kafka_exporter

bash 复制代码
# docker-compose部署多个kafka_exporter,每个exporter对接一个kafka

# cat docker-compose.yml

version: '3.1'
services:
  kafka-exporter-opslogs:
    image: bitnami/kafka-exporter:latest
    command:
      - '--kafka.server=10.2.19.43:9092'
      - '--kafka.server=10.2.24.62:9092'
      - '--kafka.server=10.5.98.190:9092'
      - '--kafka.version=3.2.1'
    restart: always
    ports:
      - 9310:9308

  kafka-exporter-prod:
    image: bitnami/kafka-exporter:latest
    command:
      - '--kafka.server=192.168.53.99:9092'
      - '--kafka.server=192.168.53.53:9092'
      - '--kafka.server=192.168.53.96:9092'
    restart: always
    ports:
      - 9311:9308

注意:配置上每个kafka broker的地址,kafka3需要指定版本

Promethus配置job接入kafka-exporter

bash 复制代码
- job_name: 'kafka-exporter'
    metrics_path: /metrics
    scrape_interval: 15s
    scrape_timeout: 10s
    static_configs:
    - targets:
      - 10.0.0.26:9310
      labels:
        name: kafka-opslogs
    - targets:
      - 10.0.0.26:9311
      labels:
        name: kafka-prod

注意:每个kafka-exporter必须增加 name标签,看板需要使用这个标签

KAFKA Grafana Dashboard

全局信息、消费者与Topic、异常与积压分析

分区维度明细

Prometheus告警规则

bash 复制代码
- name: kafka
  rules:
  - alert: KAFKA_brokers异常
    expr: kafka_broker_info != 1
    for: 2m
    labels:
      severity: critical
    annotations:
      description: "{{ $labels.name }}当前brokers异常:{{ $labels.address }}"

  - alert: 电商生产KAFKA消息整体积压
    expr: sum(kafka_consumergroup_lag_sum{job="kafka-exporter"}) by (name,consumergroup, topic)>5000
    for: 2m
    labels:
      severity: critical
    annotations:
      description: "【环境】{{ $labels.name }}\n【消费组】{{ $labels.consumergroup }}\n【topic】{{ $labels.topic }}【积压】:{{ $value | printf \"%.2f\" }}"

  - alert: 电商生产KAFKA消息分区积压
    expr: (sum(kafka_consumergroup_lag{job="kafka-exporter"}) by (name,consumergroup, topic, partition)>1500) AND ON() (hour()+8)%24 >= 7 <= 21
    for: 3m
    labels:
      severity: critical
    annotations:
      description: "【环境】{{ $labels.name }}\n【消费组】{{ $labels.consumergroup }}\n【topic】{{$labels.topic}}【分区】{{ $labels.partition }}【积压】:{{ $value | printf \"%.2f\" }}"

  - alert: 电商生产KAFKA分区数过多
    expr: sum by(name)(kafka_topic_partitions{job="kafka-exporter",topic !~"__.*"})>1500
    for: 2m
    labels:
      severity: critical
    annotations:
      description: "{{ $labels.name }}当前分区数:{{ $value | printf \"%.2f\" }}"

  - alert: 电商生产KAFKA_brokers丢失
    expr: kafka_brokers{job="kafka-exporter"} < 3
    for: 2m
    labels:
      severity: critical
    annotations:
      description: "{{ $labels.name }}当前brokers数:{{ $value | printf \"%.2f\" }}"

  - alert: 电商生产KAFKA_TopicsReplicas
    expr: sum(kafka_topic_partition_in_sync_replica{job="kafka-exporter"}) by (name,topic) <1
    for: 2m
    labels:
      severity: critical
    annotations:
      description: "{{ $labels.name }} Kafka topic in-sync partition:{{ $value | printf \"%.2f\" }}"
相关推荐
WHFENGHE5 小时前
输电线路分布式故障监测装置技术解析
分布式
nightunderblackcat6 小时前
新手向:实现验证码程序
java·spring boot·spring·java-ee·kafka·maven·intellij-idea
a587697 小时前
消息队列(MQ)高级特性深度剖析:详解RabbitMQ与Kafka
java·分布式·面试·kafka·rabbitmq·linq
hmb↑7 小时前
Kafka 3.9.x 安装、鉴权、配置详解
分布式·kafka·linq
java干货7 小时前
还在重启应用改 Topic?Spring Boot 动态 Kafka 消费的“终极形态”
spring boot·kafka·linq
lifallen7 小时前
KafkaStreams 计算图节点设计:ProcessorNode、SourceNode、SinkNode
java·数据结构·算法·kafka·apache
AAA修煤气灶刘哥8 小时前
缓存世界的三座大山:穿透、击穿、雪崩,今天就把它们铲平!
redis·分布式·后端
失散139 小时前
分布式专题——4 大厂生产级Redis高并发分布式锁实战
java·redis·分布式·缓存·架构
eqwaak010 小时前
科技信息差(9.10)
网络·人工智能·分布式·ar·智能硬件
一个帅气昵称啊10 小时前
C#,RabbitMQ从入门到精通,.NET8.0(路由/分布式/主题/消费重复问题 /延迟队列和死信队列/消息持久化 )/RabbitMQ集群模式
分布式·微服务·架构·rabbitmq·.net