etcd常用监控

通过部署etcd-exporter+Prometheus,然后配置etcd相关告警可以及时发现etcd集群风险

常见监控项目

1. etcd集群无leader

Etcd cluster have no leader

复制代码
  - alert:EtcdNoLeader
    expr: etcd_server_has_leader ==0
for:0m
    labels:
      severity: critical
    annotations:
      summary:EtcdnoLeader(instance {{ $labels.instance }})
      description: "Etcd cluster have no leader\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"```
2. etcd grpc请求慢

GRPC requests slowing down, 99th percentile is over 0.15s

复制代码
  - alert:EtcdGrpcRequestsSlow
    expr: histogram_quantile(0.99, sum(rate(grpc_server_handling_seconds_bucket{grpc_type="unary"}[1m]))by(grpc_service, grpc_method, le))>0.15
for:2m
    labels:
      severity: warning
    annotations:
      summary:Etcd GRPC requests slow (instance {{ $labels.instance }})
      description: "GRPC requests slowing down, 99th percentile is over 0.15s\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
3. etcd http请求慢

HTTP requests slowing down, 99th percentile is over 0.15s

复制代码
  - alert: EtcdHttpRequestsSlow
    expr: histogram_quantile(0.99, rate(etcd_http_successful_duration_seconds_bucket[1m])) > 0.15
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Etcd HTTP requests slow (instance {{ $labels.instance }})
      description: "HTTP requests slowing down, 99th percentile is over 0.15s\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
4. Etcd member communication slow

Etcd member communication slowing down, 99th percentile is over 0.15s

复制代码
  - alert: EtcdMemberCommunicationSlow
    expr: histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket[1m])) > 0.15
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Etcd member communication slow (instance {{ $labels.instance }})
      description: "Etcd member communication slowing down, 99th percentile is over 0.15s\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
5. Etcd high fsync durations

Etcd WAL fsync duration increasing, 99th percentile is over 0.5s

复制代码
  - alert: EtcdHighFsyncDurations
    expr: histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket[1m])) > 0.5
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Etcd high fsync durations (instance {{ $labels.instance }})
      description: "Etcd WAL fsync duration increasing, 99th percentile is over 0.5s\n  VALUE = {{ $
相关推荐
RestCloud12 小时前
SQL Server到Hive:批处理ETL性能提升30%的实战经验
数据库·api
RestCloud12 小时前
为什么说零代码 ETL 是未来趋势?
数据库·api
ClouGence14 小时前
CloudCanal + Paimon + SelectDB 从 0 到 1 构建实时湖仓
数据库
DemonAvenger21 小时前
NoSQL与MySQL混合架构设计:从入门到实战的最佳实践
数据库·mysql·性能优化
AAA修煤气灶刘哥1 天前
后端人速藏!数据库PD建模避坑指南
数据库·后端·mysql
RestCloud2 天前
揭秘 CDC 技术:让数据库同步快人一步
数据库·api
得物技术2 天前
MySQL单表为何别超2000万行?揭秘B+树与16KB页的生死博弈|得物技术
数据库·后端·mysql
可涵不会debug2 天前
【IoTDB】时序数据库选型指南:工业大数据场景下的技术突围
数据库·时序数据库
ByteBlossom2 天前
MySQL 面试场景题之如何处理 BLOB 和CLOB 数据类型?
数据库·mysql·面试
麦兜*2 天前
MongoDB Atlas 云数据库实战:从零搭建全球多节点集群
java·数据库·spring boot·mongodb·spring·spring cloud