项目小结总结

项目说明概述

该项目共分为2个子项目，由MYSQL集群高可用和数据监控平台两部分组成

MYSQL集群高可用属于云原生高级课数据库运维部分的知识

业务需求

某企业由于业务增加，超出了单库性能阈值，则需要构建高可用的数据库服务器集群，且在业务增长初期，读写业务各半，需选择Active-Active Cluster集群架构，为了保证活性，使用恰当的技术实现high availability，要求架构图如下：

为了实现自动化运维，预使用开源监控实时报警软件实现服务器的检测、大屏展示，要求使用独立主机安装软件检测数据库集群的性能，以仪表盘形式展示：

具体要求

1.网络配置：确保集群中的MySQL服务器之间的网络连接稳定可靠，避免网络延迟或丢包对集群性能造成影响。

2.同步参数配置：对于需要数据同步的集群方案（如主从复制和组复制），需要合理配置同步参数，确保数据的实时性和一致性。

3.备份和恢复策略：制定完善的备份和恢复策略，定期备份集群数据，并测试恢复流程的可行性，以确保在发生故障时能够快速恢复数据和服务。

4.监控和告警：使用监控工具对集群进行实时监控，并设置合理的告警阈值。当集群出现异常情况时，能够及时发现并处理。

项目设计思路

MYSQL集群高可用使用双主双活+keepalived实现

MYSQL数据监控平台使用mysqld_exporter+prometheus+Grafana三件套实现

集群及监控平台搭建完毕后可以实现企业内部的mysql数据库双主机在线增加高可用性，通过keepalived的故障检测和VIP漂移能力使得发生故障后使用者无感知，增加系统的容错能力，通过监控平台实现mysql数据库监控可视化

项目环境

项目拓扑结构

master1 192.168.37.167 VIP:192.167.37.200 cpu:1颗2核 内 存：2GB HDD：20GB 网 络：NAT VmWare17 OpenEuler22.03 SP4 MySql8.0.37 Keepalived2.2.4

master2 192.168.37.162 VIP:192.162.37.200 cpu:1颗2核 内 存：2GB HDD：20GB 网 络：NAT VmWare17 OpenEuler22.03 SP4 MySql8.0.37 Keepalived2.2.4

monitor 192.168.37.141 cpu:1颗2核 内 存：2GB HDD：20GB 网 络：NAT VmWare17 OpenEuler22.03 SP4 Mysqld_Exporter-0.15.1 Prometheus-2.53.2 grafana-enterprise-11.1.2

软硬件环境清单

1.环境前提配置

三台主机下载所需软件并升级

三台主机时间同步:使用阿里云时间服务器

msyql集群搭建

说明

搭建主-主集群，两个节点都是主库，也都属于对方的从库，也就是两者之间会相互同步数据，这时为了防止主键出现冲突，一般都会通过设置数据库自增步长的方式来防重

主-主架构集群适用于中读写请求的比例对半开，同时整体的并发量也不算低，至少超出了单库的承载阈值的场景下，架构图如下：

两台服务器信息如下：

|---------|----------------|-----------------------------|
| matser1 | 192.168.37.167 | OpenEuler22.03 / MySql8.4.3 |
| matser2 | 192.168.37.162 | OpenEuler22.03 / MySql8.4.3 |

master1节点配置

修改配置文件

 vim /etc/my.cnf

创建用于数据同步的账号m1

mysql> grant replication slave on *.* to 'mback'@'%';
Query OK, 0 rows affected (0.00 sec)

master2节点配置

修改配置文件

创建用于数据同步的账号m2

mysql> grant replication slave on *.* to 'mback'@'%';

Query OK, 0 rows affected (0.00 sec)

建立master1 节点主从关系

• 由于建立的是主-主架构集群，相互为对方的从节点，则两个节点都通过root账号登录

• 此时master1为从节点，master2为主节点

• 先在master2上查看同步的日志名称及同步点号

  mysql> show master status;
  +----------------------+----------+--------------+------------------+-------------------+
  | File                 | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
  +----------------------+----------+--------------+------------------+-------------------+
  | mysql-bin-log.000001 |      664 |              | mysql            |                   |
  +----------------------+----------+--------------+------------------+-------------------+
  1 row in set (0.00 sec)

  master1 节点上建立复制关系：

  mysql> change master to master_host='192.168.37.162',master_user='mback',master_password='123456',maaster_port=3306,master_log_file='mysql-bin-log.000001',master_log_pos=664;
  Query OK, 0 rows affected, 9 warnings (0.02 sec)

  mysql> start slave;
  Query OK, 0 rows affected, 1 warning (0.01 sec)

  mysql> show slave status\G
  *************************** 1. row ***************************
  Slave_IO_State: Waiting for source to send event
  Master_Host: 192.168.37.162
  Master_User: mback
  Master_Port: 3306
  Connect_Retry: 60
  Master_Log_File: mysql-bin-log.000001
  Read_Master_Log_Pos: 664
  Relay_Log_File: mysql-relay-log.000002
  Relay_Log_Pos: 330
  Relay_Master_Log_File: mysql-bin-log.000001
  Slave_IO_Running: Yes
  Slave_SQL_Running: Yes
  Replicate_Do_DB:
  Replicate_Ignore_DB:
  Replicate_Do_Table:
  Replicate_Ignore_Table:
  Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
  Last_Errno: 0
  Last_Error:
  Skip_Counter: 0
  Exec_Master_Log_Pos: 664
  Relay_Log_Space: 540
  Until_Condition: None
  Until_Log_File:
  Until_Log_Pos: 0
  Master_SSL_Allowed: No
  Master_SSL_CA_File:
  Master_SSL_CA_Path:
  Master_SSL_Cert:
  Master_SSL_Cipher:
  Master_SSL_Key:
  Seconds_Behind_Master: 0
  Master_SSL_Verify_Server_Cert: No
  Last_IO_Errno: 0
  Last_IO_Error:
  Last_SQL_Errno: 0
  Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
  Master_Server_Id: 2
  Master_UUID: 01dd4765-c6d6-11f0-acc0-000c291951ff
  Master_Info_File: mysql.slave_master_info
  SQL_Delay: 0
  SQL_Remaining_Delay: NULL
  Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates
  Master_Retry_Count: 86400
  Master_Bind:
  Last_IO_Error_Timestamp:
  Last_SQL_Error_Timestamp:
  Master_SSL_Crl:
  Master_SSL_Crlpath:
  Retrieved_Gtid_Set:
  Executed_Gtid_Set:
  Auto_Position: 0
  Replicate_Rewrite_DB:
  Channel_Name:
  Master_TLS_Version:
  Master_public_key_path:
  Get_master_public_key: 0
  Network_Namespace:
  1 row in set, 1 warning (0.00 sec)

局部测试

测试主主架构是否能相互同步数据

master1执行，master2查看

• master1上建立数据库

  mysql> create database test1;
  Query OK, 1 row affected (0.01 sec)

  mysql> show databases;
  +--------------------+
  | Database |
  +--------------------+
  | information_schema |
  | mysql |
  | performance_schema |
  | sys |
  | test1 |
  +--------------------+
  5 rows in set (0.00 sec)

• 在master2上查看是否同步

  mysql> show databases;
  +--------------------+
  | Database |
  +--------------------+
  | information_schema |
  | mysql |
  | performance_schema |
  | sys |
  | test1 |
  +--------------------+
  5 rows in set (0.01 sec)

• 此时master1(主)--master2(从)构建完毕

master2执行，master1查看

• master2中执行创建表

  mysql> use test1;
  Database changed
  mysql> create table back_test (user_idint(8) not null,user_name varchar(255) not null,user_sex varchar(255) not null,passwordd varchar(255) not null,register_time datetime);
  Query OK, 0 rows affected, 1 warning (0.02 sec)

  mysql> desc back_test;
  +---------------+--------------+------+-----+---------+-------+
  | Field | Type | Null | Key | Default | Extra |
  +---------------+--------------+------+-----+---------+-------+
  | user_id | int | NO | | NULL | |
  | user_name | varchar(255) | NO | | NULL | |
  | user_sex | varchar(255) | NO | | NULL | |
  | passwordd | varchar(255) | NO | | NULL | |
  | register_time | datetime | YES | | NULL | |
  +---------------+--------------+------+-----+---------+-------+
  5 rows in set (0.00 sec)

  mysql> exit
  Bye

• 在master1上查看是否有同步表数据

  mysql> use test1;
  Reading table information for completion of table and column names
  You can turn off this feature to get a quicker startup with -A
   
  Database changed
  mysql> desc back_test;
  +---------------+--------------+------+-----+---------+-------+
  | Field         | Type         | Null | Key | Default | Extra |
  +---------------+--------------+------+-----+---------+-------+
  | user_id       | int          | NO   |     | NULL    |       |
  | user_name     | varchar(255) | NO   |     | NULL    |       |
  | user_sex      | varchar(255) | NO   |     | NULL    |       |
  | passwordd     | varchar(255) | NO   |     | NULL    |       |
  | register_time | datetime     | YES  |     | NULL    |       |
  +---------------+--------------+------+-----+---------+-------+
  5 rows in set (0.00 sec)

至此master1(从)--master2(主)构建完毕

Keepalived故障转移的高可用环境

说明

Keepalived即保持存活，是一种轻量化的高可用方案，用来防止单点故障

Keepalived的工作原理是VRRP（Virtual Router Redundancy Protocol）虚拟路由冗余协议。

Keepalived通过VIP虚拟IP的漂移实现高可用，在相同集群内发送组播包，master1和master2主通过VRRP协议发送组播包，告诉从主的状态。

Keepalived会虚拟出一个VIP地址代理2台服务器向外界提供服务

当Keepalived实际代理master1时，若master1出现故障，keepalived通过自身机制，自动将master2作为实际代理主服务器，不需要人工干预去修改mysql配置信息，从而实现了高可用

master1节点配置

安装安装包

master1节点配置keepalived（首先检测自己的网卡名，网卡名不一样会导致IP变化

 vim /etc/keepalived/keepalived.conf

[root@master1 ~]# vim /etc/keepalived/chk_mysql.sh

 vim  /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
   router_id mysql-master02   
}
vrrp_instance VI_1 {
    state BACKUP      
    interface ens33           
    virtual_router_id 51  
    priority 50               
    nopreempt
    advert_int 1
    authentication {    
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {   
        192.168.37.200
    }
}
 
virtual_server 192.168.37.200 3306 {  
    delay_loop 6            
    lb_algo rr              
    lb_kind DR              
    persistence_timeout 50  
    protocol TCP            
    real_server 192.168.37.162 3306 {    
    notify_down /etc/keepalived/chk_mysql.sh
        weight 1        
        TCP_CHECK {
            connect_port 3306     
            connect_timeout 3     
            retry 3               
            delay_before_retry 3  
        }
    }
}

[root@master2 ~] vim  /etc/keepalived/chk_mysql.sh

[root@master2 ~]chmod  +x  /etc/keepalived/chk_mysql.sh  
[root@master2 ~]systemctl start keepalived
[root@master2 ~]systemctl status keepalived

局部测试

• 原理：当master1服务器宕机后，VIP会自动漂移至master2服务器并继续向外提供mysql服务

  [root@master1 ~]systemctl status mysql
  ● mysql.service - LSB: start and stop MySQL
  Loaded: loaded (/etc/rc.d/init.d/mysql; generated)
  Active: Active: active (running) since Fri 2025-11-21 20:37:32 CST; 24min ago

  [root@master1 ~]systemctl status keepalived
  ● keepalived.service - LVS and VRRP High Availability Monitor
  Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
  Active: active (running) since Fri 2025-11-21 20:51:50 CST; 10min ago

  [root@master1 ~]ip a
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
  link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  inet 127.0.0.1/8 scope host lo
  valid_lft forever preferred_lft forever
  inet6 ::1/128 scope host
  valid_lft forever preferred_lft forever
  2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
  link/ether 00:0c:29:da:f3:cc brd ff:ff:ff:ff:ff:ff
  inet 192.168.37.167/24 brd 192.168.88.255 scope global noprefixroute ens32
  valid_lft forever preferred_lft forever
  inet 192.168.37.200/32 scope global ens33
  valid_lft forever preferred_lft forever
  inet6 fe80::20c:29ff:feda:f3cc/64 scope link noprefixroute
  valid_lft forever preferred_lft forever

  [root@master1 ~]systemctl stop mysql

  [root@master1 ~] ip a
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
  link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  inet 127.0.0.1/8 scope host lo
  valid_lft forever preferred_lft forever
  inet6 ::1/128 scope host
  valid_lft forever preferred_lft forever
  2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
  link/ether 00:0c:29:da:f3:cc brd ff:ff:ff:ff:ff:ff
  inet 192.168.37.167/24 brd 192.168.88.255 scope global noprefixroute ens33
  valid_lft forever preferred_lft forever
  inet6 fe80::20c:29ff:feda:f3cc/64 scope link noprefixroute
  valid_lft forever preferred_lft forever

  [root@master1 ~]systemctl status keepalived
  ○ keepalived.service - LVS and VRRP High Availability Monitor
  Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
  Active: inactive (dead)

master2执行

[root@master2 ~] ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:19:51:ff brd ff:ff:ff:ff:ff:ff
    inet 192.168.37.162/24 brd 192.168.37.255 scope global dynamic noprefixroute ens33
       valid_lft 1726sec preferred_lft 1726sec
    inet 192.168.37.200/32 scope global ens33     #VIP显示
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe19:51ff/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

master1中恢复msyql服务和keepalived服务，注意必须先恢复msyql服务

[root@master1 ~]systemctl start mysql
[root@master1 ~]systemctl start keepalived
[root@master1 ~]ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:6a:a9:23 brd ff:ff:ff:ff:ff:ff
    inet 192.168.37.167/24 brd 192.168.37.255 scope global dynamic noprefixroute ens33
       valid_lft 1249sec preferred_lft 1249sec
    inet6 fe80::20c:29ff:fe6a:a923/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

安装监控平台

说明

• 涉及的软件及端口

|-----------------|------|
| 服务 | 端口 |
| Prometheus | 9090 |
| Mysqld_exporter | 9194 |
| Grafana | 3000 |

部署Mysqld_exporter

• Mysql_exporter是用来收集MysQL数据库相关指标且需要连接到数据库并有相关权限。

• 下载安装包并解压缩：

  192.168.37.141 monitor主机操作

  [root@monitor ~] ls
  anaconda-ks.cfg mysqld_exporter-0.15.1.linux-amd64.tar.gz
  [root@monitor ~] tar xvf mysqld_exporter-0.15.1.linux-amd64.tar.gz
  [root@monitor ~]mv mysqld_exporter-0.15.1.linux-amd64 /usr/local/mysqld_exporter
  [root@monitor ~] cd /usr/local/mysqld_exporter
  [root@monitor mysqld_exporter]# ls
  LICENSE mysqld_exporter NOTICE

• 常见配置文件

  [root@monitor mysqld_exporter]# vim  .my.cnf
  [client]
  user = exporter         
  password = 123456     
  host = 192.168.37.200    
  port = 3306

• 2台mysql节点创建用户并授权

  [root@master1 ~] mysql -uroot -p
  Enter password:
  mysql> create user 'exporter'@'%' identified with mysql_native_password by '123456';
  Query OK, 0 rows affected (0.01 sec)
  mysql> grant process, replication client, select on . to 'exporter'@'%';
  Query OK, 0 rows affected (0.00 sec)
  mysql> flush privileges;
  Query OK, 0 rows affected (0.01 sec)
  mysql> exit
  [root@master2 ~]# mysql -uroot -p
  Enter password:
  mysql> select user ,host from mysql.user;
  +------------------+-----------+
  | user | host |
  +------------------+-----------+
  | exporter | % | # 已经同步
  | mback | % |
  | mysql.infoschema | localhost |
  | mysql.session | localhost |
  | mysql.sys | localhost |
  | root | localhost |
  +------------------+-----------+
  6 rows in set (0.00 sec)
  mysql> exit

• 配置mysqld_exporter的系统服务

  [root@monitor ~]vim /usr/lib/systemd/system/mysqld_exporter.service
  [Unit]
  Description=https://prometheus.io
   
  [Service]
  Restart=on-failure
  ExecStart=/usr/local/mysqld_exporter/mysqld_exporter --config.my-cnf=/usr/local/mysqld_exporter/.my.cnf --web.listen-address=:9104
   
  [Install]
  WantedBy=multi-user.target

• 刷新服务配置并启动服务

  [root@monitor ~]systemctl daemon-reload
  [root@monitor ~]systemctl start mysqld_exporter.service
  [root@monitor ~]systemctl enable mysqld_exporter.service
  [root@monitor ~]systemctl status mysqld_exporter.service

• 通过浏览器输入http://192.168.37.141:9104/metrics，查看监控抓取msyql服务器的数据

部署Promethue

Prometheus是一个开源系统监控和警报工具包，最初由SoundCloud构建。自2012年成立以来，许多公司和组织都采用了Prometheus，该项目拥有非常活跃的开发人员和用户社区。它现在是一个独立的开源项目，独立于任何公司维护。

Prometheus于2016年加入了云原生计算基金会，成为继Kubernetes之后的第二个托管项目。

架构图：

[root@monitor ~]tar xvf prometheus-2.53.2.linux-amd64.tar.gz
 
[root@monitor ~]mv prometheus-2.53.2.linux-amd64  /usr/local/prometheus
 
[root@monitor ~]cd /usr/local/prometheus
[root@monitor prometheus]# ls
console_libraries  consoles  LICENSE  NOTICE  prometheus  prometheus.yml  promtool

新建prometheus账户

[root@monitor ~]useradd --no-create-home --shell /bin/false prometheus
[root@monitor ~]chown -R prometheus:prometheus /usr/local/prometheu

配置prometheus系统服务

[root@monitor ~] vim  /usr/lib/systemd/system/prometheus.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
 
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/prometheus/prometheus --config.file /usr/local/prometheus/prometheus.yml --storage.tsdb.path  /usr/local/prometheus/data
 
[Install]
WantedBy=multi-user.target

刷新服务配置并启动服务

[root@monitor ~]systemctl  daemon-reload
[root@monitor ~]systemctl enable  --now  prometheus
[root@monitor ~]systemctl  status  prometheus

通过浏览器输入http://192.168.37.141:9090，查看管理页面，通过http://192.168.37.141:9090/metrics 查看监控数据

Mysqld_exporter对接Prometheus

[root@monitor ~]vim  /usr/local/prometheus/prometheus.yml
 
 - job_name: "mysql"
    static_configs:
      - targets: ["192.168.37.163:9104"] 
        labels:
          instance: mysqld_exporter

部署Grafana

Grafana是一个功能强大、灵活性高、易于使用的数据可视化和监控工具，广泛应用于IT运维、应用性能监控、工业物联网等领域

使用Grafana，用户可以轻松地创建各种图表、图形和面板，以直观和动态的方式展示数据趋势、指标和警报。其灵活的插件系统和丰富的图形化选项使用户能够根据自己的需求定制仪表板，并将其集成到现有的监控系统中

下载并解压缩：

修改配置文件

[root@monitor ~]# vim /etc/grafana/grafana.ini

启动

[root@monitor ~]systemctl enable --now  grafana-server
[root@monitor ~]systemctl status grafana-server

关联Promethues并设置仪表盘模版

浏览器其中输入：http://192.168.37.141:3000

初始账户/密码：admin/admin

设置新的密码：123456

在设置界面中选择Home > Connections > Data sources > prometheus

设置prometheus监听地址：http://192.168.37.141:9090

save&test

选择监控模版来显示mysql的关键指标，模版ID为7362

压力测试

• 查看VIP所在服务器，停止当前节点的mysql，查看VIP是否漂移，刷新监控界面，看是否高可用

• MySQL自带的压力测试工具------Mysqlslap

  vim /etc/my.cnf
  max_connections=1024

  systemctl restart mysql

  完成压测，查看仪表盘

  mysqlslap --defaults-file=/etc/my.cnf --concurrency=200,400 --iterations=1 --number-int-cols=50 --number-char-cols=60 --auto-generate-sql --auto-generate-sql-add-autoincrement --auto-generate-sql-load-type=mixed --engine=innodb --number-of-queries=2000 -uroot -p123456 --verbose

结束项目