ProxySQL实现mysql8主从同步读写分离

ProxySQL基本介绍

ProxySQL是 MySQL 的高性能、高可用性、协议感知代理。以下为结合主从复制对ProxySQL读写分离、黑白名单、路由规则等做些基本测试。

先简单介绍下ProxySQL及其功能和配置，主要包括：

最基本的读/写分离，且方式有多种；

可定制基于用户、基于schema、基于语句的规则对SQL语句进行路由，规则很灵活；

动态加载配置，即绝大部分的配置可以在线修改，但有少部分参数还是需要重启来生效；

可缓存查询结果。虽然缓存策略比较简陋，但实现了基本的缓存功能；

过滤危险的SQL，增加防火墙等功能；

提供连接池、日志记录、审计日志等功能；

请求流程

核心功能

读写分离：可查询走从库，写入走主库

简单Sharding：ProxySQL的sharding是通过正则匹配来实现的，对于需要拆分SQL以及合并SQL执行结果的不能支持，所以写了简单sharding

连接池管理：常规功能，为了提高SQL执行效率。

多路复用：主要优化点在后端mysql连接的复用，对比smart client，中间层不仅对前端建连也会对后端建连，可自行控制后端连接的复用逻辑。

流量管控：kill连接和kill query；whitelist配置。

高可用：底层mysql，如果从库挂了，自动摘除流量；主库挂了暂不处理。proxysql自身高可用，提供cluster的功能，cluster内部会自行同步元数据以及配置变更信息。

查询缓存：对username+schema+query的key进行缓存，设置ttl过期，不适合写完就查的场景，因为在数据在未过期之前可能是脏数据。

动态配置：大部分的配置可动态变更，先load到runtime，在save到disk，通过cluster的功能同步到其他的节点。

流量镜像：同一份流量可以多出写入，但是并不保证mirror的流量一定成功。

ProxySQL结构

Qurey Processor 用于匹配查询规则并根据规则决定是否缓存查询或者将查询加入黑名单或者重新路由、重写查询或者镜像查询到其他hostgroup。
User Auth 为底层后端数据库认证提供了用户凭证。
Hostgroup manager -- 负责管理发送SQL请求都后端数据库并跟踪SQL请求状态。
Connection pool -- 负责管理后端数据库连接，连接池中建立的连接被所有的前端应用程序共享。
Monitoring -- 负责监控后端数据库健康状态主从复制延时并临时下线不正常的数据库实例。

数据库结构

sql 复制代码

# mysql -uadmin -padmin -h127.0.0.1 -P6032
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.5.30 (ProxySQL Admin Module)
mysql> show databases;
+-----+---------------+-------------------------------------+
| seq | name          | file                                |
+-----+---------------+-------------------------------------+
| 0   | main          |                                     |
| 2   | disk          | /var/lib/proxysql/proxysql.db       |
| 3   | stats         |                                     |
| 4   | monitor       |                                     |
| 5   | stats_history | /var/lib/proxysql/proxysql_stats.db |
+-----+---------------+-------------------------------------+
5 rows in set (0.00 sec)

main：内存配置数据库，表里存放后端db实例、用户验证、路由规则等信息。表名以 runtime开头的表示proxysql当前运行的配置内容，不能通过dml语句修改，只能修改对应的不以 runtime 开头的（在内存）里的表，然后 LOAD 使其生效， SAVE 使其存到硬盘以供下次重启加载。
disk：是持久化到硬盘的配置，sqlite数据文件。SQLite3 数据库，默认位置为 $（DATADIR）/proxysql.db,在重新启动时，未保留的内存中配置将丢失。因此，将配置保留在 DISK 中非常重要。(SQLite是一个进程内的库，实现了自给自足的、无服务器的、零配置的、事务性的 SQL 数据库引擎)
stats：proxysql运行抓取的统计信息，包括到后端各命令的执行次数、流量、processlist、查询种类汇总/执行时间等等。
monitor：库存储 monitor 模块收集的信息，主要是对后端db的健康/延迟检查。
stats_history:统计信息历史库

实验环境

机器名称	IP配置	服务角色	备注
proxy	192.168.142.146	proxysql控制器	用于监控管理
master	192.168.142.147	数据库主服务器
slave1	192.168.142.139	数据库从服务器

安装ProxySQL

bash 复制代码

# 配置yum源
cat >/etc/yum.repos.d/proxysql.repo << EOF
[proxysql]
name=ProxySQL YUM repository
baseurl=https://repo.proxysql.com/ProxySQL/proxysql-2.5.x/centos/8
gpgcheck=1
gpgkey=https://repo.proxysql.com/ProxySQL/proxysql-2.5.x/repo_pub_key
EOF
c
# 安装proxysql
yum install -y proxysql

启动 ProxySQL

bash 复制代码

[root@master ~]# systemctl enable --now proxysql

# 管理员登录
[root@master ~]# mysql -uadmin -padmin -h 127.0.0.1 -P 6032
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.5.30 (ProxySQL Admin Module)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]> show databases;
+-----+---------------+-------------------------------------+
| seq | name          | file                                |
+-----+---------------+-------------------------------------+
| 0   | main          |                                     |
| 2   | disk          | /var/lib/proxysql/proxysql.db       |
| 3   | stats         |                                     |
| 4   | monitor       |                                     |
| 5   | stats_history | /var/lib/proxysql/proxysql_stats.db |
+-----+---------------+-------------------------------------+
5 rows in set (0.00 sec)

可见有五个库： main、disk、stats 、monitor 和 stats_history
main: 内存配置数据库，即 MEMORY，表里存放后端 db 实例、用户验证、路由规则等信息。main 库中有如下信息：

MySQL [(none)]> show tables from main;
+----------------------------------------------------+
| tables                                             |
+----------------------------------------------------+
| coredump_filters                                   |
| global_variables                                   |
| mysql_aws_aurora_hostgroups                        |
| mysql_collations                                   |
| mysql_firewall_whitelist_rules                     |
| mysql_firewall_whitelist_sqli_fingerprints         |
| mysql_firewall_whitelist_users                     |
| mysql_galera_hostgroups                            |
| mysql_group_replication_hostgroups                 |
| mysql_hostgroup_attributes                         |
| mysql_query_rules                                  |
| mysql_query_rules_fast_routing                     |
| mysql_replication_hostgroups                       |
| mysql_servers                                      |
| mysql_users                                        |
| proxysql_servers                                   |
| restapi_routes                                     |
| runtime_checksums_values                           |
| runtime_coredump_filters                           |
| runtime_global_variables                           |
| runtime_mysql_aws_aurora_hostgroups                |
| runtime_mysql_firewall_whitelist_rules             |
| runtime_mysql_firewall_whitelist_sqli_fingerprints |
| runtime_mysql_firewall_whitelist_users             |
| runtime_mysql_galera_hostgroups                    |
| runtime_mysql_group_replication_hostgroups         |
| runtime_mysql_hostgroup_attributes                 |
| runtime_mysql_query_rules                          |
| runtime_mysql_query_rules_fast_routing             |
| runtime_mysql_replication_hostgroups               |
| runtime_mysql_servers                              |
| runtime_mysql_users                                |
| runtime_proxysql_servers                           |
| runtime_restapi_routes                             |
| runtime_scheduler                                  |
| scheduler                                          |
+----------------------------------------------------+
36 rows in set (0.00 sec)

库下的主要表：
mysql_servers: 后端可以连接 MySQL 服务器的列表
mysql_users: 配置后端数据库的账号和监控的账号。
mysql_query_rules: 指定 Query 路由到后端不同服务器的规则列表。

注： 表名以 runtime_开头的表示 ProxySQL 当前运行的配置内容，不能通过 DML 语句修改。

只能修改对应的不以 runtime 开头的表，然后 "LOAD" 使其生效，"SAVE" 使其存到硬盘以供下次重启加载。
disk ：持久化的磁盘的配置
stats： 统计信息的汇总
monitor：一些监控的收集信息，比如数据库的健康状态等
stats_history: 这个库是 ProxySQL 收集的有关其内部功能的历史指标

配置 ProxySQL 所需账户

在 Master (192.168.142.146) 的MySQL 上创建 ProxySQL 的监控账户和对外访问账户

bash 复制代码

create user 'monitor'@'192.168.%.%' identified with mysql_native_password by 'Monitor@123.com';
grant all privileges on *.* to 'monitor'@'192.168.%.%' with grant option;

#proxysql 的对外访问账户
create user 'proxysql'@'192.168.%.%' identified with mysql_native_password by '123456';
grant all privileges on *.* to 'proxysql'@'192.168.%.%' with grant option;

配置ProxySQL

配置ProxySQL主从分组信息

bash 复制代码

MySQL [(none)]>
*************************** 1. row ***************************
       table: mysql_replication_hostgroups
Create Table: CREATE TABLE mysql_replication_hostgroups (
    writer_hostgroup INT CHECK (writer_hostgroup>=0) NOT NULL PRIMARY KEY,
    reader_hostgroup INT NOT NULL CHECK (reader_hostgroup<>writer_hostgroup AND reader_hostgroup>=0),
    check_type VARCHAR CHECK (LOWER(check_type) IN ('read_only','innodb_read_only','super_read_only','read_only|innodb_read_only','read_only&innodb_read_only')) NOT NULL DEFAULT 'read_only',
    comment VARCHAR NOT NULL DEFAULT '', UNIQUE (reader_hostgroup))
1 row in set (0.00 sec)

创建组：(定义写为1，读为0)

sql 复制代码

MySQL [(none)]> insert into mysql_replication_hostgroups (writer_hostgroup,reader_hostgroup,comment) values (1,0,'proxy');
Query OK, 1 row affected (0.00 sec)

MySQL [(none)]> load mysql servers to runtime;
Query OK, 0 rows affected (0.01 sec)

MySQL [(none)]> save mysql servers to disk;
Query OK, 0 rows affected (0.02 sec)

注意：ProxySQL会根据server的read_only的取值将服务器进行分组。read_only=0的server，master被分到编号为1的写组，read_only=1的server，slave则分到编号为0的读组

MySQL [(none)]> select * from mysql_replication_hostgroups;
+------------------+------------------+------------+---------+
| writer_hostgroup | reader_hostgroup | check_type | comment |
+------------------+------------------+------------+---------+
| 1                | 0                | read_only  | proxy   |
+------------------+------------------+------------+---------+
1 row in set (0.00 sec)

添加主从服务器节点：

sql 复制代码

写
MySQL [(none)]> insert into mysql_servers(hostgroup_id,hostname,port) values (1,'192.168.142.147',3306);
Query OK, 1 row affected (0.00 sec)
读1
MySQL [(none)]> insert into mysql_servers(hostgroup_id,hostname,port) values (0,'192.168.142.139',3306);
Query OK, 1 row affected (0.00 sec)
MySQL [(none)]> load mysql servers to runtime;
Query OK, 0 rows affected (0.01 sec)

MySQL [(none)]> save mysql servers to disk;
Query OK, 0 rows affected (0.00 sec)

MySQL [(none)]> select * from mysql_servers;
+--------------+----------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname       | port | gtid_port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+----------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1            | 192.168.142.147 | 3306 | 0         | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 0            | 192.168.142.139 | 3306 | 0         | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 0            | 192.168.150.23 | 3306 | 0         | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
+--------------+----------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
3 rows in set (0.00 sec)

为ProxySQL监控MySQL后端节点

sql 复制代码

MySQL [(none)]> use monitor
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MySQL [monitor]> set mysql-monitor_username='monitor';
Query OK, 1 row affected (0.00 sec)

MySQL [monitor]> set mysql-monitor_password='Monitor@123.com';
Query OK, 1 row affected (0.00 sec)
MySQL [monitor]> load mysql variables to runtime;
MySQL [monitor]> save mysql variables to disk;
MySQL [(none)]> select @@mysql-monitor_username;
+--------------------------+
| @@mysql-monitor_username |
+--------------------------+
| monitor                  |
+--------------------------+
1 row in set (0.00 sec)
MySQL [(none)]> select @@mysql-monitor_password;
+--------------------------+
| @@mysql-monitor_password |
+--------------------------+
| Monitor@123.com          |
+--------------------------+
1 row in set (0.00 sec)
Admin>select * from monitor.mysql_server_connect_log;
+-----------------+------+------------------+-------------------------+--------------------------------------------------------------------------+
| hostname        | port | time_start_us    | connect_success_time_us | connect_error                                                            |
+-----------------+------+------------------+-------------------------+--------------------------------------------------------------------------+
| 192.168.142.139 | 3306 | 1709457246014919 | 0                       | NULL |
| 192.168.142.147 | 3306 | 1709457246735181 | 1543                    | NULL                                                                     |
| 192.168.142.139 | 3306 | 1709457306015351 | 0                       | NULL |
| 192.168.142.147 | 3306 | 1709457306922075 | 1232                    | NULL                                                                     |
| 192.168.142.147 | 3306 | 1709457366016055 | 1555                    | NULL                                                                     |

对心跳信息的监控：

sql 复制代码

Admin>select * from mysql_server_ping_log limit 10;
+-----------------+------+------------------+----------------------+--------------------------------------------------------------------------+
| hostname        | port | time_start_us    | ping_success_time_us | ping_error                                                               |
+-----------------+------+------------------+----------------------+--------------------------------------------------------------------------+
| 192.168.142.147 | 3306 | 1709457295768656 | 238                  | NULL                                                                     |
| 192.168.142.139 | 3306 | 1709457295768744 | 0                    | NULL |

查看read_only日志监控：

sql 复制代码

Admin>select * from mysql_server_read_only_log limit 5;
+-----------------+------+------------------+-----------------+-----------+--------------------------------------------------------------------------------------------------------------+
| hostname        | port | time_start_us    | success_time_us | read_only | error                                                                                                        |
+-----------------+------+------------------+-----------------+-----------+--------------------------------------------------------------------------------------------------------------+
| 192.168.142.147 | 3306 | 1709457334932365 | 449             | 0         | NULL                                                                                                         |
| 192.168.142.139 | 3306 | 1709457334932475 | 0               | NULL      | NULL |
| 192.168.142.147 | 3306 | 1709457336432467 | 1110            | 0         | NULL

ProxySQL配置对外访问账号：

bash 复制代码

MySQL [(none)]> show create table mysql_users\G
*************************** 1. row ***************************
       table: mysql_users
Create Table: CREATE TABLE mysql_users (
    username VARCHAR NOT NULL,
    password VARCHAR,
    active INT CHECK (active IN (0,1)) NOT NULL DEFAULT 1,
    use_ssl INT CHECK (use_ssl IN (0,1)) NOT NULL DEFAULT 0,
    default_hostgroup INT NOT NULL DEFAULT 0,
    default_schema VARCHAR,
    schema_locked INT CHECK (schema_locked IN (0,1)) NOT NULL DEFAULT 0,
    transaction_persistent INT CHECK (transaction_persistent IN (0,1)) NOT NULL DEFAULT 1,
    fast_forward INT CHECK (fast_forward IN (0,1)) NOT NULL DEFAULT 0,
    backend INT CHECK (backend IN (0,1)) NOT NULL DEFAULT 1,
    frontend INT CHECK (frontend IN (0,1)) NOT NULL DEFAULT 1,
    max_connections INT CHECK (max_connections >=0) NOT NULL DEFAULT 10000,
    attributes VARCHAR CHECK (JSON_VALID(attributes) OR attributes = '') NOT NULL DEFAULT '',
    comment VARCHAR NOT NULL DEFAULT '',
    PRIMARY KEY (username, backend),
    UNIQUE (username, frontend))
1 row in set (0.00 sec)

将对外访问账号添加到mysql_users表中：

sql 复制代码

insert into mysql_users (username,password,default_hostgroup,transaction_persistent) values ('proxysql','123456',1,1);
 
load mysql users to runtime;
save mysql users to disk;
MySQL [(none)]> select * from mysql_users\G

在从库端192.168.142.139上通过对方访问账号proxy连接，测试是否路由能默认到hostgroup_id=1，它是一个写组

sql 复制代码

[root@salve2 ~]# mysql -h192.168.142.146 -uproxysql -p'123456' -P 6033
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.5.30 (ProxySQL)

Copyright (c) 2000, 2024, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| itcast             |
| master1db          |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
6 rows in set (0.01 sec)
mysql>  select @@server_id;
+-------------+
| @@server_id |
+-------------+
|           1 |
+-------------+
1 row in set (0.00 sec)

mysql>  create database keme;
Query OK, 1 row affected (0.01 sec)

添加读写分离规则(mysql_query_rules)

sql 复制代码

Admin>insert into mysql_query_rules(rule_id,active,match_pattern,destination_hostgroup,apply) values(1,1,'^select .* for update$',1,1);
Query OK, 1 row affected (0.00 sec)

Admin>insert into mysql_query_rules(rule_id,active,match_pattern,destination_hostgroup,apply) values(2,1,'^select',0,1);
Query OK, 1 row affected (0.00 sec)

Admin>load mysql query rules to runtime;
Query OK, 0 rows affected (0.00 sec)

Admin>save mysql query rules to disk;
Query OK, 0 rows affected (0.01 sec)

测试读写分离

sql 复制代码

[root@salve2 ~]#  mysql -uproxysql -p123456 -h 192.168.142.147 -P 3306 -e "select @@server_id"
mysql: [Warning] Using a password on the command line interface can be insecure.
+-------------+
| @@server_id |
+-------------+
|           1 |
+-------------+