5 MySQL-主从复制&分库分表
5.1mysql 主从复制
5.1.1. 概述
主从复制是将主数据库的DDL和DML操作通过二进制日志(binlog文件)传送到从库服务器,然后在从库上对这些日志重新执行,从而使得主库和从库的数据保持同步。
MySQL支持一台主库同时向多台从库进行复制,从库也可以从其他从库复制。(主从,从从等模式)。
MySQL复制的优点包含以下几个方面:
- 主库出现问题,可以快速切换到从库提供服务。
- 实现读写分离,降低主库的访问压力。(增删改操作在主库,读取在从库)
- 可以在从库中执行备份(因为加了全局锁,锁了库),以避免备份期间影响主库服务。
5.1.2. 原理
基于binlog实现主从模式。
从库有两组线程,IO Thread和SQL Thread
IO Thread复制连接到主库,并且读取主库的binlog日志,并且将其保存到从节点的中继日志,即relay日志。SQL Thread复制读取relay日志,并且执行。
5.1.3. 搭建
基于Docker实现MySQL主从复制(全网最详细!!!)_docker mysql-CSDN博客
5.1.3.1. 搭建主节点

powershell
docker run -d \
--name mysql_master \
-p 3307:3306 \
-e MYSQL_ROOT_PASSWORD=123456 \
-v /usr/local/dlh/mysql_master/data:/var/lib/mysql \
-v /usr/local/dlh/mysql_master/conf:/etc/mysql \
-v /usr/local/dlh/mysql_master/mysql-files:/var/lib/mysql-files \
-v /usr/local/dlh/mysql_master/log:/var/log/mysql \
mysql:8.0.23
!注意修改MYSQL的配置文件!
powershell
[mysqld]
# 服务器唯一id,保证主从集群环境中唯一,取值范围:1~(2^32-1),默认为1
server-id=1
# 是否只读,1代表只读,0代表读写
read-only=0
# 二进制日志名,默认binlog
# log-bin=binlog
# 设置需要复制的数据库,默认复制全部数据库
# binlog-do-db=mytestdb
# 设置不需要复制的数据库
# binlog-ignore-db=mysql
# binlog-ignore-db=infomation_schema
skip-name-resolve
5.1.3.1.1. 创建主从用户
sql
登录mysql,创建远程连接的账号,并授予主从复制权限。
docker exec -it mysql_master /bin/bash
mysql -uroot -p123456
# 创建remote_user用户,并且指定其密码
# 该用户可以在任意的主机连接该MYSQL服务
create user 'remote_user'@'%'
identified with mysql_native_password by 'Root@123456';
或者:
CREATE USER 'remote_user'@'%' IDENTIFIED BY 'Root@123456';
# 给'remote_user'@'%' 用户分配主从复制权限
grant replication slave on *.* to 'remote_user'@'%';
查看二进制日志坐标(binlog和offset):
sql
show master status;
+---------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+---------------+----------+--------------+------------------+-------------------+
| binlog.000002 | 673 | | | |
+---------------+----------+--------------+------------------+-------------------+
- file:从哪个日志文件开始推送日志文件
- position:文件日志偏移量
5.1.3.2. 搭建从节点
powershell
docker run -d \
--name mysql_slave \
-p 3308:3306 \
-e MYSQL_ROOT_PASSWORD=123456 \
-v /usr/local/dlh/mysql_slave/log:/var/log/mysql \
-v /usr/local/dlh/mysql_slave/data:/var/lib/mysql \
-v /usr/local/dlh/mysql_slave/conf:/etc/mysql \
-v /usr/local/dlh/mysql_slave/mysql-files:/var/lib/mysql-files \
mysql:8.0.23
添加从节点配置文件。
powershell
[mysqld]
# 服务器唯一id,保证主从集群环境中唯一,取值范围:1~(2^32-1),默认为1
server-id=2
# 是否只读,1代表只读,0代表读写
read-only=1
# 二进制日志名,默认binlog
# log-bin=binlog
# 设置需要复制的数据库,默认复制全部数据库
# binlog-do-db=mytestdb
# 设置不需要复制的数据库
# binlog-ignore-db=mysql
# binlog-ignore-db=infomation_schema
skip-name-resolve
5.1.3.2.1. 从库关联到主库
sql
#进入从库
docker exec -it mysql_slave /bin/bash
mysql -uroot -p123456
#与主库连接
CHANGE MASTER TO MASTER_HOST='192.168.171.108',
MASTER_USER='remote_user',MASTER_PASSWORD='Root@123456', MASTER_PORT=3307,
MASTER_LOG_FILE='binlog.000005',MASTER_LOG_POS=673;

5.1.3.2.2. 开启主从同步
sql
start replica;
输入后,查看从库状态。
sql
show replica status\G
# 输出:
******************** 1. row ***************************
Replica_IO_State: Waiting for master to send event
Source_Host: 192.168.171.108
Source_User: remote_user
Source_Port: 3307
Connect_Retry: 60
Source_Log_File: binlog.000002
Read_Source_Log_Pos: 673
Relay_Log_File: 1edd24b6c96d-relay-bin.000002
Relay_Log_Pos: 321
Relay_Source_Log_File: binlog.000002
Replica_IO_Running: Yes
Replica_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Source_Log_Pos: 673
Relay_Log_Space: 537
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Source_SSL_Allowed: No
Source_SSL_CA_File:
Source_SSL_CA_Path:
Source_SSL_Cert:
Source_SSL_Cipher:
Source_SSL_Key:
Seconds_Behind_Source: 0
Source_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Source_Server_Id: 1
Source_UUID: 26e38018-4127-11f0-ab8e-0242ac110002
Source_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Replica_SQL_Running_State: Slave has read all relay log; waiting for more updates
Source_Retry_Count: 86400
Source_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Source_SSL_Crl:
Source_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name:
Source_TLS_Version:
Source_public_key_path:
Get_Source_public_key: 0
Network_Namespace:
注意:如果发现IO线程或者SQL线程未启用;可执行以下命令重试:
sql
在从库上执行 STOP SLAVE;
使用 RESET SLAVE; 命令清除现有的复制信息。
然后重新设置从库的主库信息,使用 CHANGE MASTER TO ... 命令,并提供正确的日志文件名和位置。
最后,执行 START SLAVE; 尝试重新开始复制。
5.1.4. shadingSphere实战演示
5.1.4.1简介
ShardingSphere 是一个开源的分布式数据库中间件,提供数据分片、分布式事务和数据库治理功能。其核心原理包括 SQL 解析、路由、改写、执行和归并等多个步骤。
详细可参考文档:ShardingSphere内核原理及核心源码剖析
5.1.4.2 Springboot集成shadingSphere实现读写分离
properties
spring.application.name=sharding
# 数据库名
spring.shardingsphere.datasource.names = mysql_master,mysql_slave
# 数据库详细信息
spring.shardingsphere.datasource.mysql_master.type = com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.mysql_master.driver-class-name = com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.mysql_master.url = jdbc:mysql://192.168.171.108:3307/test?characterEncoding=UTF-8&useSSL=false
spring.shardingsphere.datasource.mysql_master.username = root
spring.shardingsphere.datasource.mysql_master.password = 123456
spring.shardingsphere.datasource.mysql_slave.type = com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.mysql_slave.driver-class-name = com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.mysql_slave.url = jdbc:mysql://192.168.171.108:3308/test?characterEncoding=UTF-8&useSSL=false
spring.shardingsphere.datasource.mysql_slave.username = root
spring.shardingsphere.datasource.mysql_slave.password = 123456
# 路由规则
spring.shardingsphere.rules.readwrite-splitting.data-sources.readwrite_ds.static-strategy.write-data-source-name = mysql_master
spring.shardingsphere.rules.readwrite-splitting.data-sources.readwrite_ds.static-strategy.read-data-source-names = mysql_master,mysql_slave
# 打印sql
spring.shardingsphere.props.sql-show=true
xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.dlh</groupId>
<artifactId>sharding</artifactId>
<version>1.0-SNAPSHOT</version>
<name>sharding</name>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.7.7</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<properties>
<java.version>11</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.apache.shardingsphere</groupId>
<artifactId>shardingsphere-jdbc-core-spring-boot-starter</artifactId>
<version>5.2.1</version>
</dependency>
<dependency>
<groupId>com.baomidou</groupId>
<artifactId>mybatis-plus-boot-starter</artifactId>
<version>3.5.7</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.16</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.yaml</groupId>
<artifactId>snakeyaml</artifactId>
<version>1.33</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
5.1.4. 总结

5.2分库&分表
5.2.1介绍

如果大数据量进行数据存储,存在以下性能瓶颈:
- IO瓶颈:请求太多,导致大量的磁盘IO,效率太低。请求的数据太多,带宽不够,网络IO平静。
- CPU瓶颈:排序,分组,连接查询,聚合统计等SQL会消耗大量的CPU资源,请求数太多,CPU出现瓶颈。
5.2.2拆分策略
分库分表只是粒度层面的不同,垂直拆分和水平拆分只是在拆分维度上的区别。
- 分库就是将一个数据库分成多个数据库。
- 分表就是将一个表拆分成多个表。
5.2.2.1垂直分库&垂直分表

5.2.2.2水平拆分

5.2.3垂直分库
5.2.3.1搭建docker环境

搭建mysql_server1
shell
docker run -d \
--name mysql_server1 \
-p 3309:3306 \
-e MYSQL_ROOT_PASSWORD=123456 \
-v /usr/local/dlh/mysql_server1/data:/var/lib/mysql \
-v /usr/local/dlh/mysql_server1/conf:/etc/mysql \
-v /usr/local/dlh/mysql_server1/mysql-files:/var/lib/mysql-files \
-v /usr/local/dlh/mysql_server1/log:/var/log/mysql \
mysql:8.0.23
搭建mysql_server2
powershell
docker run -d \
--name mysql_server2 \
-p 3310:3306 \
-e MYSQL_ROOT_PASSWORD=123456 \
-v /usr/local/dlh/mysql_server2/data:/var/lib/mysql \
-v /usr/local/dlh/mysql_server2/conf:/etc/mysql \
-v /usr/local/dlh/mysql_server2/mysql-files:/var/lib/mysql-files \
-v /usr/local/dlh/mysql_server2/log:/var/log/mysql \
mysql:8.0.23
在mysql-server1 服务器上, 创建数据库 payorder_db,并创建表 pay_order
sql
CREATE DATABASE msb_payorder_db CHARACTER SET 'utf8';
CREATE TABLE pay_order (
order_id bigint(20) NOT NULL AUTO_INCREMENT,
user_id int(11) DEFAULT NULL, -- 表关联
product_name varchar(128) DEFAULT NULL,
COUNT int(11) DEFAULT NULL,
PRIMARY KEY (order_id)
) ENGINE=InnoDB AUTO_INCREMENT=12345679 DEFAULT CHARSET=utf8
在mysql-server2 服务器上, 创建数据库 msb_user_db,并创建表users
sql
CREATE DATABASE user_db CHARACTER SET 'utf8';
CREATE TABLE users (
id int(11) NOT NULL,
username varchar(255) NOT NULL COMMENT '用户昵称',
phone varchar(255) NOT NULL COMMENT '注册手机',
PASSWORD varchar(255) DEFAULT NULL COMMENT '用户密码',
PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='用户表'
5.2.3.2配置文件
使用sharding-jdbc 对数据库中水平拆分的表进行操作,通过sharding-jdbc对分库分表的规则进行配置,配置内容包括:数据源、主键生成策略、分片策略等。
sql
spring.application.name=sharding_2
# 定义多个数据源
spring.shardingsphere.datasource.names = mysql-server1,mysql-server2
# datasource 1
spring.shardingsphere.datasource.mysql-server1.type = com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.mysql-server1.driver-class-name = com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.mysql-server1.url = jdbc:mysql://192.168.31.145:3309/payorder_db?characterEncoding=UTF-8&useSSL=false
spring.shardingsphere.datasource.mysql-server1.username = root
spring.shardingsphere.datasource.mysql-server1.password = 123456
# datasource 2
spring.shardingsphere.datasource.mysql-server2.type = com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.mysql-server2.driver-class-name = com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.mysql-server2.url = jdbc:mysql://192.168.31.145:3310/user_db?characterEncoding=UTF-8&useSSL=false
spring.shardingsphere.datasource.mysql-server2.username = root
spring.shardingsphere.datasource.mysql-server2.password = 123456
# 配置规则
spring.shardingsphere.rules.sharding.tables.pay_order.actual-data-nodes=mysql-server1.pay_order
spring.shardingsphere.rules.sharding.tables.users.actual-data-nodes=mysql-server2.users
# print sql
spring.shardingsphere.props.sql-show=true
配置规则格式:
测试是否垂直分库成功。
java
@SpringBootTest
class Sharding2ApplicationTests {
@Resource
private PayOrderMapper payOrderMapper;
@Resource
private UsersMapper usersMapper;
@Test
public void testInsert(){
Users user = new Users();
user.setId(1002);
user.setUsername("大远哥");
user.setPhone("15612344321");
user.setPassword("123456");
usersMapper.insert(user);
PayOrder payOrder = new PayOrder();
payOrder.setOrderId(12345679L);
payOrder.setProductName("猕猴桃");
payOrder.setUserId(user.getId());
payOrder.setCount(2);
payOrderMapper.insert(payOrder);
}
}
5.2.4水平分表
如果单表数据量越来越多的时候,对一张表的操作性能将会下降。所以我们将一张表水平拆分成多个表,分散在不同的数据库中存储。
场景:用户订单表pay_order中数据量太多,如何实现水平分表?
5.2.4.1数据准备
powershell
docker run -d \
--name mysql_server3 \
-p 3311:3306 \
-e MYSQL_ROOT_PASSWORD=123456 \
-v /usr/local/dlh/sharding/mysql_server3/log:/var/log/mysql \
-v /usr/local/dlh/sharding/mysql_server3/data:/var/lib/mysql \
-v /usr/local/dlh/sharding/mysql_server3/conf:/etc/mysql \
-v /usr/local/dlh/sharding/mysql_server3/mysql-files:/var/lib/mysql-files \
mysql:8.0.23

需求说明:
- 在mysql-server3服务器上, 创建数据库 course_db
- 创建表 t_course_1 、 t_course_2
- 约定规则:如果添加的课程 id 为偶数添加到 t_course_1 中,奇数添加到 t_course_2 中。
水平分片的id需要在业务层实现,不能依赖数据库的主键自增
sql
CREATE TABLE t_course_1 (
`cid` BIGINT(20) NOT NULL,
`user_id` BIGINT(20) DEFAULT NULL,
`cname` VARCHAR(50) DEFAULT NULL,
`brief` VARCHAR(50) DEFAULT NULL,
`price` DOUBLE DEFAULT NULL,
`status` INT(11) DEFAULT NULL,
PRIMARY KEY (`cid`)
) ENGINE=INNODB DEFAULT CHARSET=utf8
CREATE TABLE t_course_2 (
`cid` BIGINT(20) NOT NULL,
`user_id` BIGINT(20) DEFAULT NULL,
`cname` VARCHAR(50) DEFAULT NULL,
`brief` VARCHAR(50) DEFAULT NULL,
`price` DOUBLE DEFAULT NULL,
`status` INT(11) DEFAULT NULL,
PRIMARY KEY (`cid`)
) ENGINE=INNODB DEFAULT CHARSET=utf8
5.2.4.2修改配置文件
sql
spring.shardingsphere.datasource.names = mysql-server3
spring.shardingsphere.datasource.mysql-server3.type = com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.mysql-server3.driver-class-name = com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.mysql-server3.url = jdbc:mysql://192.168.31.145:3309/course_db?characterEncoding=UTF-8&useSSL=false
spring.shardingsphere.datasource.mysql-server3.username = root
spring.shardingsphere.datasource.mysql-server3.password = 123456
5.2.4.3数据节点配置
修改t_course表的实际节点映射。(即表在哪个节点上)
sql
spring.shardingsphere.rules.sharding.tables.t_course.actual-data-nodes=master-server3.t_course_1
spring.shardingsphere.rules.sharding.tables.t_course.actual-data-nodes=master-server3.t_course_2
如果实际节点太多,那么将会写很多次。可以用下面的方式简化。
行表达式的使用: 行表达式
powershell
spring.shardingsphere.rules.sharding.tables.t_course.actual-data-nodes=mysql-server3.t_course_$->{1..2}
spring.shardingsphere.rules.sharding.tables.t_course.actual-data-nodes=master-server3.t_course_$->{1..2}
表达式 db1.t_course_$->{1..2}
会被 大括号中的 `{1..2}` 所替换, `{begin..end}` 表示范围区间
会有两种选择: master-server3.t_course_1 和 master-server3.t_course_2
5.2.4.4 配置分片策略
分片策略包括分片键和分片算法.
分片规则,约定cid值为偶数时,添加到t_course_1表,如果cid是奇数则添加到t_course_2表
分片算法比如取模,hash等。
powershell
#1.配置数据节点
#指定course表的分布情况(配置表在哪个数据库,表名是什么)
spring.shardingsphere.rules.sharding.tables.t_course.actual-data-nodes=db1.t_course_$->{1..2}
##2.配置分片策略(分片策略包括分片键和分片算法)
#2.1 分片键名称: cid
spring.shardingsphere.rules.sharding.tables.t_course.table-strategy.standard.sharding-column=cid
#2.2 分片算法名称
spring.shardingsphere.rules.sharding.tables.t_course.table-strategy.standard.sharding-algorithm-name=table-inline
#2.3 分片算法类型: 行表达式分片算法(标准分片算法下包含->行表达式分片算法)
spring.shardingsphere.rules.sharding.sharding-algorithms.table-inline.type=INLINE
#2.4 分片算法属性配置
spring.shardingsphere.rules.sharding.sharding-algorithms.table-inline.props.algorithm-expression=t_course_$->{cid % 2 + 1}
5.2.4.5 测试
测试插入:
java
@Test
public void testInsertCourse(){
for (int i = 1; i < 10; i++) {
Course course = new Course();
course.setCid(1L+i);
course.setUserId(1L+i);
course.setCname("Java经典面试题讲解");
course.setBrief("课程涵盖目前最容易被问到的10000道Java面试题");
course.setPrice(100.0);
course.setStatus(1);
courseMapper.insert(course);
}
}
测试查询:
java
@Test
public void test3(){
List<Course> courses = courseMapper.selectList(null);
for (Course cours : courses) {
System.out.println(cours);
}
}
测试分页查询:
java
@Test
public void test4() {
//全量如何分页?
List<Course> courses1 = courseMapper.selestectPage(1, 5);
for (Course c : courses1) {
System.out.println(c);
}
}