etcd常用监控

通过部署etcd-exporter+Prometheus,然后配置etcd相关告警可以及时发现etcd集群风险

常见监控项目

1. etcd集群无leader

Etcd cluster have no leader

复制代码
  - alert:EtcdNoLeader
    expr: etcd_server_has_leader ==0
for:0m
    labels:
      severity: critical
    annotations:
      summary:EtcdnoLeader(instance {{ $labels.instance }})
      description: "Etcd cluster have no leader\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"```
2. etcd grpc请求慢

GRPC requests slowing down, 99th percentile is over 0.15s

复制代码
  - alert:EtcdGrpcRequestsSlow
    expr: histogram_quantile(0.99, sum(rate(grpc_server_handling_seconds_bucket{grpc_type="unary"}[1m]))by(grpc_service, grpc_method, le))>0.15
for:2m
    labels:
      severity: warning
    annotations:
      summary:Etcd GRPC requests slow (instance {{ $labels.instance }})
      description: "GRPC requests slowing down, 99th percentile is over 0.15s\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
3. etcd http请求慢

HTTP requests slowing down, 99th percentile is over 0.15s

复制代码
  - alert: EtcdHttpRequestsSlow
    expr: histogram_quantile(0.99, rate(etcd_http_successful_duration_seconds_bucket[1m])) > 0.15
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Etcd HTTP requests slow (instance {{ $labels.instance }})
      description: "HTTP requests slowing down, 99th percentile is over 0.15s\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
4. Etcd member communication slow

Etcd member communication slowing down, 99th percentile is over 0.15s

复制代码
  - alert: EtcdMemberCommunicationSlow
    expr: histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket[1m])) > 0.15
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Etcd member communication slow (instance {{ $labels.instance }})
      description: "Etcd member communication slowing down, 99th percentile is over 0.15s\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
5. Etcd high fsync durations

Etcd WAL fsync duration increasing, 99th percentile is over 0.5s

复制代码
  - alert: EtcdHighFsyncDurations
    expr: histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket[1m])) > 0.5
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Etcd high fsync durations (instance {{ $labels.instance }})
      description: "Etcd WAL fsync duration increasing, 99th percentile is over 0.5s\n  VALUE = {{ $
相关推荐
九皇叔叔2 小时前
【9】PostgreSQL 之 vacuum 死元组清理
数据库·postgresql
风雅的远行者2 小时前
mysql互为主从失效,重新同步
数据库·mysql
宇钶宇夕3 小时前
S7-1200 系列 PLC 中 SCL 语言的 PEEK 和 POKE 指令使用详解
运维·服务器·数据库·程序人生·自动化
绿蚁新亭3 小时前
Spring的事务控制——学习历程
数据库·学习·spring
scilwb4 小时前
占用栅格地图数据集
数据库
时序数据说5 小时前
时序数据库的存储之道:从数据特性看技术要点
大数据·数据库·物联网·开源·时序数据库·iotdb
鸥梨菌Honevid5 小时前
QT解析文本框数据——概述
数据库·qt·mysql
今天又得骑车了6 小时前
一、MySQL 8.0 之《EXPLAIN ANALYZE 执行计划》
数据库·mysql·database
野犬寒鸦7 小时前
MyBatis-Plus 中使用 Wrapper 自定义 SQL
java·数据库·后端·sql·mybatis
我爱一条柴ya7 小时前
【AI大模型】RAG系统组件:向量数据库(ChromaDB)
数据库·人工智能·pytorch·python·ai·ai编程