etcd常用监控

通过部署etcd-exporter+Prometheus,然后配置etcd相关告警可以及时发现etcd集群风险

常见监控项目

1. etcd集群无leader

Etcd cluster have no leader

复制代码
  - alert:EtcdNoLeader
    expr: etcd_server_has_leader ==0
for:0m
    labels:
      severity: critical
    annotations:
      summary:EtcdnoLeader(instance {{ $labels.instance }})
      description: "Etcd cluster have no leader\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"```
2. etcd grpc请求慢

GRPC requests slowing down, 99th percentile is over 0.15s

复制代码
  - alert:EtcdGrpcRequestsSlow
    expr: histogram_quantile(0.99, sum(rate(grpc_server_handling_seconds_bucket{grpc_type="unary"}[1m]))by(grpc_service, grpc_method, le))>0.15
for:2m
    labels:
      severity: warning
    annotations:
      summary:Etcd GRPC requests slow (instance {{ $labels.instance }})
      description: "GRPC requests slowing down, 99th percentile is over 0.15s\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
3. etcd http请求慢

HTTP requests slowing down, 99th percentile is over 0.15s

复制代码
  - alert: EtcdHttpRequestsSlow
    expr: histogram_quantile(0.99, rate(etcd_http_successful_duration_seconds_bucket[1m])) > 0.15
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Etcd HTTP requests slow (instance {{ $labels.instance }})
      description: "HTTP requests slowing down, 99th percentile is over 0.15s\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
4. Etcd member communication slow

Etcd member communication slowing down, 99th percentile is over 0.15s

复制代码
  - alert: EtcdMemberCommunicationSlow
    expr: histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket[1m])) > 0.15
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Etcd member communication slow (instance {{ $labels.instance }})
      description: "Etcd member communication slowing down, 99th percentile is over 0.15s\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
5. Etcd high fsync durations

Etcd WAL fsync duration increasing, 99th percentile is over 0.5s

复制代码
  - alert: EtcdHighFsyncDurations
    expr: histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket[1m])) > 0.5
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Etcd high fsync durations (instance {{ $labels.instance }})
      description: "Etcd WAL fsync duration increasing, 99th percentile is over 0.5s\n  VALUE = {{ $
相关推荐
Mr.Entropy4 小时前
请求超过Spring线程池的最大线程(处理逻辑)
数据库·sql·spring
GBASE4 小时前
“G”术时刻:南大通用GBase 8c数据库权限管理场景实践(二)
数据库
wearegogog1235 小时前
MySQL中实施排序(sorting)及分组(grouping)操作
数据库·mysql
2301_803554525 小时前
MySQL 主从读写分离架构
数据库·mysql·架构
正在走向自律5 小时前
Ubuntu系统下Python连接国产KingbaseES数据库实现增删改查
开发语言·数据库·python·ubuntu·kingbasees·ksycopg2
没有bug.的程序员6 小时前
Redis 内存管理机制:深度解析与性能优化实践
java·数据库·redis·性能优化·内存管理机制
小蒜学长6 小时前
基于SpringBoot+Vue的健身房管理系统的设计与实现(代码+数据库+LW)
java·数据库·vue.js·spring boot·后端
失散136 小时前
分布式专题——2 深入理解Redis线程模型
java·数据库·redis·分布式·架构
DemonAvenger6 小时前
数据库迁移实战:最小化停机时间的方法与经验分享
数据库·sql·性能优化
xhbh6667 小时前
Python操作MySQL的两种姿势:原生SQL与ORM框架SQLAlchemy详解
数据库·oracle