Prometheus监控MySQL主从数据库

案例分析Prometheus监控MySQL主从数据库

1. 规划节点

节点规划，见表1。

表1 节点规划

IP	主机名	节点
192.168.100.3	master	Prometheus + Grafana
192.168.100.3	master	mysql/master + mysqld_exporter + node_exporter
192.168.100.4	node	mysql/master + mysqld_exporter + node_exporter

案例实施

1. 基础环境准备

（1）配置节点时间同步

[root@master ~]# yum install -y chrony
[root@master ~]# systemctl enable chronyd --now
[root@master ~]# vim /etc/chrony.conf
...
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
#server 192.168.100.3 iburst
server ntp1.aliyun.com iburst
server ntp2.aliyun.com iburst
server ntp3.aliyun.com iburst
...
# Allow NTP client access from local network.
#allow 192.168.0.0/16
allow 192.168.100.0/16
...

（2）重启服务

[root@master ~]# systemctl restart chronyd

（3）同步阿里云时间服务器

[root@master ~]# chronyc sources
210 Number of sources = 2
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^- 120.25.115.20                 2   6    25     2  +3773us[+3773us] +/-   26ms
^* 203.107.6.88                  2   6    17    10  +1189us[+4744us] +/-   19ms
[root@master ~]# date
Thu Sep 26 08:39:22 CST 2024

2. 配置MySQL主从

（1）配置 yum 源

[root@master ~]# vim /etc/yum.repos.d/centos.repo 
[centos]
name=centos
baseurl=file:///opt/centos
gpgcheck=0
[k8s]
name=k8s
baseurl=file:///opt/kubernetes-repo
gpgcheck=0

（2）安装 mariadb 服务

master节点安装

[root@master ~]# yum install -y mariadb-server mariadb

启动服务

[root@master ~]# systemctl enable mariadb --now

设置mysql登录密码

[root@master ~]# mysqladmin password 000000

master节点配置 my.cnf 文件

[root@master ~]# vim /etc/my.cnf
[mysqld]
log_bin=mysql-bin
server_id=1
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

重启服务

[root@master ~]# systemctl restart mariadb

node节点安装

[root@node ~]# yum install -y mariadb-server mariadb

启动服务

[root@node ~]# systemctl enable mariadb --now

设置mysql登录密码

[root@node ~]# mysqladmin password 000000

node节点配置 my.cnf 文件

[root@node ~]# vim /etc/my.cnf
[mysqld]
log_bin=mysql-bin
server_id=2
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

重启服务

[root@node ~]# systemctl restart mariadb

（3）sql语句配置主从

master节点

[root@master ~]# mysql -uroot -p000000
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 3
Server version: 5.5.68-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> grant replication slave on *.* to 'slave'@'%' identified by '000000';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> grant replication slave on *.* to 'slave'@'localhost' identified by '000000';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.00 sec)

node节点

[root@node ~]# mysql -uroot -p000000
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 6
Server version: 5.5.68-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> CHANGE MASTER TO
    -> MASTER_HOST='192.168.100.3',
    -> MASTER_USER='slave',
    -> MASTER_PASSWORD='000000';
Query OK, 0 rows affected (0.01 sec)

MariaDB [(none)]> start slave;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.100.3
                  Master_User: slave
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000001
          Read_Master_Log_Pos: 752
               Relay_Log_File: mariadb-relay-bin.000002
                Relay_Log_Pos: 1036
        Relay_Master_Log_File: mysql-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 752
              Relay_Log_Space: 1332
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 1
1 row in set (0.01 sec)

3. 编写启动项（方法一）

（1）安装 Prometheus

[root@master ~]# tar -zxvf prometheus-2.37.0.linux-amd64.tar.gz -C /usr/local/
[root@master ~]# mv /usr/local/prometheus-2.37.0.linux-amd64 /usr/local/prometheus
[root@master ~]# cd /usr/lib/systemd/system/
[root@master system]# cat prometheus.service
[Unit]
Description=prometheus

[Service]
ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target

重新加载 Systemd 配置文件

[root@master system]# systemctl daemon-reload
[root@master system]# systemctl enable prometheus --now
[root@master system]# netstat -ntpl | grep 9090

（2）安装 Grafana

[root@master ~]# tar -zxvf grafana-enterprise-8.3.6.linux-amd64.tar.gz -C /usr/local/
[root@master ~]# cd /usr/lib/systemd/system/
[root@master system]# cat grafana.service
[Unit]
Description=grafana

[Service]
ExecStart=/usr/local/grafana-8.3.6/bin/grafana-server -homepath=/usr/local/grafana-8.3.6/
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target

重新加载 Systemd 配置文件

[root@master system]# systemctl daemon-reload
[root@master system]# systemctl enable grafana --now
[root@master ~]# netstat -ntpl | grep 3000

（3）安装 Node Exporter

[root@master ~]# scp node_exporter-1.3.1.linux-amd64.tar.gz mysqld_exporter-0.12.1.linux-amd64.tar.gz master:/root/
[root@master ~]# scp node_exporter-1.3.1.linux-amd64.tar.gz mysqld_exporter-0.12.1.linux-amd64.tar.gz node:/root/
[root@master ~]# tar -zxvf node_exporter-1.3.1.linux-amd64.tar.gz -C /usr/local/
[root@master ~]# cd /usr/lib/systemd/system
[root@master system]# cat node_exporter.service
[Unit]
Description=node_exporter

[Service]
ExecStart=/usr/local/node_exporter-1.3.1.linux-amd64/node_exporter
Restart=always

[Install]
WantedBy=multi-user.target

重新加载 Systemd 配置文件

[root@master system]# systemctl daemon-reload
[root@master system]# systemctl enable node_exporter --now
[root@master system]# netstat -ntpl | grep node

（4）编写启动项

[root@master ~]# cd /usr/lib/systemd/system
[root@master system]# cat mysqld_exporter.service
[Unit]
Description=mysqld_exporter

[Service]
ExecStart=/usr/local/mysqld_exporter-0.12.1.linux-amd64/mysqld_exporter --config.my-cnf=/usr/local/mysqld_exporter-0.12.1.linux-amd64/my.cnf
Restart=always

[Install]
WantedBy=multi-user.target

重新加载 Systemd 配置文件

[root@master system]# systemctl daemon-reload
[root@master system]# systemctl enable mysqld_exporter --now
[root@master system]# netstat -ntpl | grep mysqld

4. 二进制启动（方法二）

（1）启动 Prometheus

[root@master ~]# tar -zxvf prometheus-2.37.0.linux-amd64.tar.gz -C /usr/local/
[root@master ~]# cd /usr/local/prometheus-2.37.0.linux-amd64/
[root@master prometheus-2.37.0.linux-amd64]# nohup ./prometheus &  
[root@master prometheus-2.37.0.linux-amd64]# netstat -ntpl | grep 9090

（2）启动 Grafana

[root@master ~]# tar -zxvf grafana-enterprise-8.3.6.linux-amd64.tar.gz -C /usr/local/
[root@master ~]# cd /usr/local/grafana-8.3.6/bin
[root@master bin]# nohup ./grafana-server &  
[root@master bin]# netstat -ntpl | grep 3000

（3）启动 Node Exporter

[root@master ~]# tar -zxvf node_exporter-1.3.1.linux-amd64.tar.gz -C /usr/local/
[root@master ~]# cd /usr/local/node_exporter-1.3.1.linux-amd64/
[root@master node_exporter-1.3.1.linux-amd64]# nohup ./node_exporter &  
[root@master node_exporter-1.3.1.linux-amd64]# netstat -ntpl | grep 9100

（4）安装 mysqld_exporter

[root@master ~]# tar -zvxf mysqld_exporter-0.12.1.linux-amd64.tar.gz -C /usr/local/
[root@master ~]# cd /usr/local/mysqld_exporter-0.12.1.linux-amd64/
[root@master mysqld_exporter-0.14.0.linux-amd64]# cat my.cnf
[client]
user=root
password='123456'
[root@master mysqld_exporter-0.14.0.linux-amd64]# ./mysqld_exporter --config.my-cnf=./my.cnf 

5. 配置 Prometheus 监控

（1）编写监控系统的配置文件

[root@master ~]# cd /usr/local/prometheus/
[root@master prometheus]# cat prometheus.yml
# my global config
global:
  scrape_interval: 5s
  evaluation_interval: 5s

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# Scrape configurations
scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
  
  - job_name: "mysql-master-slave"
    static_configs:
      - targets: ["192.168.100.3:9104"]
      - targets: ["192.168.100.4:9104"]
        labels:
          instance: mysql-master-slave
  
  - job_name: "nodes"
    static_configs:
      - targets: ["192.168.100.3:9100"]
      - targets: ["192.168.100.4:9100"]
        labels:
          instance: nodes
  - job_name: "grafana"
    static_configs:
      - targets: ["192.168.100.3:3000"]
        labels:
          instance: grafana

（2）重启 Prometheus 后刷新页面

# 启动项方式部署
[root@master system]# systemctl restart prometheus
# 二进制启动
[root@master prometheus-2.37.0.linux-amd64]# netstat -ntpl | grep prometheus
tcp6       0      0 :::9090                 :::*                    LISTEN      128942/./prometheus 
[root@master prometheus-2.37.0.linux-amd64]# pkill -9 prometheus
[root@master prometheus-2.37.0.linux-amd64]# ./prometheus

（3）登录到 Prometheus 控制台查看状态

如果出现报错 out of bounds 删除数据目录

[root@master prometheus-2.37.0.linux-amd64]# ll
total 206628
drwxr-xr-x 2 3434 3434        38 Jul 14  2022 console_libraries
drwxr-xr-x 2 3434 3434       173 Jul 14  2022 consoles
drwxr-xr-x 8 root root       206 Sep 26 07:00 data
-rw-r--r-- 1 3434 3434     11357 Jul 14  2022 LICENSE
-rw------- 1 root root    434968 Sep 25 19:07 nohup.out
-rw-r--r-- 1 3434 3434      3773 Jul 14  2022 NOTICE
-rwxr-xr-x 1 3434 3434 109655433 Jul 14  2022 prometheus
-rw-r--r-- 1 3434 3434      1175 Sep 25 19:11 prometheus.yml
-rwxr-xr-x 1 3434 3434 101469395 Jul 14  2022 promtool
[root@master prometheus-2.37.0.linux-amd64]# rm -rf data/

再次启动 Prometheus

[root@master prometheus-2.37.0.linux-amd64]# ./prometheus

（4）配置 Grafana 监控模版

（5）配置 Grafana Dashboard

（6）查看 MySQL 状态

监控模板7371

（7）查看 Node 状态

监控模版12227