Prometheus告警统计（好用）-开源项目alertsnitch

开源项目

https://github.com/yakshaving-art/alertsnitch

Grafana成果展示

1）告警级别分类统计

2）具体故障时间点、恢复时间点内容记录

支持loki存储版

https://github.com/MikeHsu0618/alertsnitch

前言

alertsnitch开源项目，统计alertmanager的告警次数

流程：**alertmanager -> webhook服务 -> mysql -> grafana **

目前生产的最佳实践，severity配置如下

P1级别，critical -> 电话，重要紧急，立即处理
P2级别，warning -> 钉钉群，重要不紧急，xx分钟内处理
P3级别，info -> 钉钉群，不重要不紧急，视情况处理

一、已有MySQL情况下部署 alertsnitch

1. MySQL准备（如果没有mysql则参考文末）

SQL 复制代码

1、创建用户、数据库、授权
create database alertsnitch;
use alertsnitch;
# mysql8.x
ALTER USER 'alertsnitch_Admin'@'%' IDENTIFIED WITH mysql_native_password BY '密码';

# MariaDB 10.6.11
ALTER USER 'alertsnitch_Admin'@'%' IDENTIFIED VIA mysql_native_password USING PASSWORD('密码');
GRANT ALL ON alertsnitch.* TO 'alertsnitch_Admin'@'%';

2、初始化SQL
use alertsnitch;
然后执行下边2个sql  # mysql默认存储/var/lib/mysql，如果mysql是docker启动的话，可以将数据cp到这个目录，然后source xxx.sql
https://github.com/yakshaving-art/alertsnitch/blob/master/db.d/mysql/0.0.1-bootstrap.sql
https://github.com/yakshaving-art/alertsnitch/blob/master/db.d/mysql/0.1.0-fingerprint.sql

2. 启动alertsnitch服务

YAML 复制代码

# vim docker-compose.yml
services:
  alertsnitch:
    image: "registry.gitlab.com/yakshaving.art/alertsnitch"
    ports:
      - "9567:9567"
    environment:
      ALERTSNITCH_DEBUG: true
      ALERTSNITCH_BACKEND: "mysql"
      ALERTSNITCH_DSN: "alertsnitch_Admin:密码@tcp(IP地址:3306)/alertsnitch"
    restart: always
    deploy:
      resources:
        limits:
          cpus: "0.5"
          memory: "512M"

docker-compose up -d

3. 配置Alertmanager

YAML 复制代码

# 原有配置可以不动，在原先上追加，等于多发webhook一份
route:
    - receiver: 'alertsnitch'
      matchers:
        - severity=~"info|warning|critical"
      continue: true

receivers:
  - name: 'alertsnitch'
    webhook_configs:
      - url: 'http://localhost:9567/webhook'

重启alertmanager

4. Grafana 添加MySQL数据源

略

5. 导入报表

15833

二、直通部署情况（mysql+alertsnitch 放在docker-compose）

Bash 复制代码

mkdir -p /data/ops/alertsnitch
cd /data/ops/alertsnitch
mkdir -p db.d/mysql mysql-data

将sql 下载下来放到 db.d/mysql中

https://github.com/yakshaving-art/alertsnitch/blob/master/db.d/mysql/0.0.1-bootstrap.sql

https://github.com/yakshaving-art/alertsnitch/blob/master/db.d/mysql/0.1.0-fingerprint.sql

YAML 复制代码

services:
  alertsnitch:
    image: registry.gitlab.com/yakshaving.art/alertsnitch:0.2
    ports:
    - "9567:9567"
    environment:
      ALERTSNITCH_DSN: "alertsnitch:root_alertsnitch@tcp(mysqldb)/alertsnitch"
      ALERSTNITCH_BACKEND: "mysql"
    depends_on:
      mysqldb:
        condition: service_healthy
    restart: always
    deploy:
      resources:
        limits:
          cpus: "0.5"
          memory: "512M"

  mysqldb:
    restart: always
    image: "mysql:5.7"
    command: "mysqld"
    volumes:
      - ./mysql-data:/var/lib/mysql
      - ./db.d/mysql:/docker-entrypoint-initdb.d
    ports:
    - "3306:3306"
    environment:
      MYSQL_DATABASE: alertsnitch
      MYSQL_USER: "alertsnitch"
      MYSQL_PASSWORD: "root_alertsnitch"
      MYSQL_ROOT_PASSWORD: "root_alertsnitch"
    healthcheck:
      test: ["CMD", "mysqladmin" ,"ping", "-h", "localhost", "-proot_alertsnitch"]
      timeout: 20s
      retries: 10

启动服务

docker-compose up -d

三、Loki数据源部署模式

alertsnitch 切换环境变量配置 + 切换镜像（另一个开源项目）

YAML 复制代码

version: '3.7'

services:
  alertsnitch:
    image: mikehsu0618/alertsnitch:latest
    ports:
      - "9567:9567"
    environment:
      ALERTSNITCH_DEBUG: true
      ALERTSNITCH_BACKEND_ENPOINT: "http://10.180.200.17:38727"
      ALERTSNITCH_BACKEND: "loki"
    depends_on:
      - "loki"

docker-compose up -d