sharding proxy 实战读写分离，分库分表

1. 简介

text 复制代码

Sharding-Proxy 是一种数据库中间件，通常用于分布式数据库的架构中，主要用于实现数据库的分片（Sharding）功能。它的作用是对外提供统一的数据库访问接口，并在后端根据数据的分片规则，将请求路由到具体的数据库分片上。

具体来说，Sharding-Proxy 的作用包括：

- 透明的分片管理：Sharding-Proxy 通过配置规则，自动将查询请求分发到相应的数据库分片上。应用程序不需要感知数据库分片的存在，它提供了一个统一的数据库访问接口。

- 分布式查询：Sharding-Proxy 支持分布式查询的执行，能够在多个分片之间进行数据的联合查询，并将查询结果合并后返回给客户端。

- 负载均衡：Sharding-Proxy 可以根据分片的负载情况来智能地分配请求，避免某个分片过载，从而提升系统的性能和可扩展性。

- 高可用性：Sharding-Proxy 支持故障转移和容错机制，确保数据库的高可用性。当某个分片节点出现故障时，可以自动切换到其他可用节点，避免单点故障。

- 支持多种数据库：Sharding-Proxy 通常支持多种数据库类型，比如 MySQL、PostgreSQL 等，可以将不同类型的数据库进行统一管理。

- 数据路由和事务管理：Sharding-Proxy 负责根据数据的分片规则路由请求，同时能够管理跨多个分片的事务，保证数据一致性。

2. 准备工作

sharding proxy 以及mysql数据库采用docker容器安装，默认都会用docker-compose安装使用。

数据库准备：

服务器	服务器 IP	数据库	表名
server1	192.168.21.130	tc_test_0	sys_user
server1	192.168.21.130	tc_test_1	sys_user
server2	192.168.21.131	tc_test_0	sys_user
server2	192.168.21.131	tc_test_1	sys_user

目标：

server1作为主数据库服务器
server2作为从数据库服务器
根据表sys_user表字段user_id进行分片，存储在不同的库内,分别为tc_test_0和tc_test_1

3. 数据库安装

1. 主数据库安装

docker-compose 安装命令：

yaml 复制代码

mysql:
    image: mysql:8.0.33
    container_name: mysql
    environment:
      # 时区上海
      TZ: Asia/Shanghai
      # root 密码
      MYSQL_ROOT_PASSWORD: 123456
      # 初始化数据库
      MYSQL_DATABASE: test
    ports:
      - "3306:3306"
    volumes:
      # 数据挂载
      - /home/mysql/data/:/var/lib/mysql/
      # 配置挂载
      - /home/mysql/conf/:/etc/mysql/conf.d/
    command:
      --default-authentication-plugin=mysql_native_password
      --character-set-server=utf8mb4
      --collation-server=utf8mb4_general_ci
      --explicit_defaults_for_timestamp=true
      --lower_case_table_names=1
    privileged: true
    network_mode: "host"

在挂载目录内修改并创建my.cnf配置文件

tex 复制代码

vi /home/mysql/conf/my.cnf

配置文件内容

sql 复制代码

[mysqld]
server-id=1
binlog_format=MIXED
# 二进制日志名 默认binlog
# log-bin=binlog
# 设置要复制的数据库
# binlog-do-db=mytestdb
# 设置不需要复制的数据库
# binlog-ignore-db=mysql

重启数据库：

java 复制代码

docker restart [容器id]

主服务器设置：

sql 复制代码

-- 创建slave用户
CREATE USER 'mysql_slave'@'%';
-- 设置密码
ALTER USER 'mysql_slave'@'%' IDENTIFIED WITH mysql_native_password BY '123456';
-- 授予复制权限
GRANT REPLICATION SLAVE ON *.* 'mysql_slave'@'%';
-- 刷新权限
FLUSH PRIVILEGES;

主机中查询master状态：

执行完此步骤后不要再操作主服务MYSQL，防止主服务器状态值变化

sql 复制代码

SHOW MASTER STATUS;

记下File和Position的值，执行完此步骤后不要再操作主服务器MYSQL，防止主服务器状态值变化。

2. 从数据库安装

docker-compose 安装命令：

sql 复制代码

mysql:
    image: mysql:8.0.33
    container_name: mysql
    environment:
      # 时区上海
      TZ: Asia/Shanghai
      # root 密码
      MYSQL_ROOT_PASSWORD: 123456
      # 初始化数据库
      MYSQL_DATABASE: test
    ports:
      - "3306:3306"
    volumes:
      # 数据挂载
      - /home/mysql/data/:/var/lib/mysql/
      # 配置挂载
      - /home/mysql/conf/:/etc/mysql/conf.d/
    command:
      --default-authentication-plugin=mysql_native_password
      --character-set-server=utf8mb4
      --collation-server=utf8mb4_general_ci
      --explicit_defaults_for_timestamp=true
      --lower_case_table_names=1
    privileged: true
    network_mode: "host"

在挂载目录内修改并创建my.cnf配置文件

tex 复制代码

vi /home/mysql/conf/my.cnf

配置文件内容

sql 复制代码

[mysqld]
# 重要字段，不能同主服务器一样
server-id=2
# 错误日志忽略 1062 主键重复 1452 外键约束等
# slave-skip-errors=1032,1062,1452

重启数据库

tex 复制代码

docker restart [容器id]

从服务器配置主从关系：

sql 复制代码

CHANGE MASTER TO MASTER_HOST='192.168.21.130',MASTER_USER='mysql_slave',MASTER_PASSWORD='123456',MASTER_PORT=3306,MASTER_LOG_FILE='binlog.000031',MASTER_LOG_POS=8701878;

启动主从同步：

启动从机的复制功能，执行sql：

sql 复制代码

START SLAVE;
-- 查看状态
SHOW SLAVE STATUS\G

下面两个参数都是Yes，则说明配置成功：

4. 测试主从复制

在主机创建目标数据库以及表：

sql 复制代码

-- 创建tc_test_0
CREATE DATABASE tc_test_0;
USE tc_test_0;
CREATE TABLE sys_user (
	user_id BIGINT AUTO_INCREMENT,
    name VARCHAR(30),
    PRIMARY KEY (user_id)
);
-- 创建tc_test_1
CREATE DATABASE tc_test_1;
USE tc_test_1;
CREATE TABLE sys_user (
	user_id BIGINT AUTO_INCREMENT,
    name VARCHAR(30),
    PRIMARY KEY (user_id)
);

查看从库，从库已经创建两个数据库以及对应表结构。

5. sharding proxy 实现读写分离，分库分表

在192.168.21.130上安装，docker-compose安装命令：

java 复制代码

shardingproxy:
    image: apache/shardingsphere-proxy:5.4.0
    container_name: shardingsphere-proxy
    command: server /data
    ports:
      - "3307:3307"
    volumes:
      - /docker/shardingproxy/conf:/opt/shardingsphere-proxy/conf
      - /docker/shardingproxy/ext-lib:/opt/shardingsphere-proxy/ext-lib
    environment:
      - JVM_OPTS="-Djava.awt.headless=true"
    network_mode: "host"

修改conf文件夹下面的配置文件

yaml 复制代码

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

######################################################################################################
#
# If you want to configure governance, authorization and proxy properties, please refer to this file.
#
######################################################################################################

# mode:
#  type: Cluster
#  repository:
#    type: ZooKeeper
#    props:
#      namespace: governance_ds
#      server-lists: localhost:2181
#      retryIntervalMilliseconds: 500
#      timeToLiveSeconds: 60
#      maxRetries: 3
#      operationTimeoutMilliseconds: 500

authority:
 users:
   # 授权用户名
   - user: root@%
   # 授权用户密码
     password: root
 privilege:
   type: ALL_PERMITTED

transaction:
 defaultType: XA
 providerType: Atomikos

sqlParser:
 sqlCommentParseEnabled: false
 sqlStatementCache:
   initialCapacity: 2000
   maximumSize: 65535
 parseTreeCache:
   initialCapacity: 128
   maximumSize: 1024

logging:
 loggers:
 - loggerName: ShardingSphere-SQL
   additivity: true
   level: INFO
   props:
     enable: false

sqlFederation:
  sqlFederationEnabled: true
  executionPlanCache:
    initialCapacity: 2000
    maximumSize: 65535

props:
 system-log-level: INFO
 max-connections-size-per-query: 1
 kernel-executor-size: 16  # Infinite by default.
 proxy-frontend-flush-threshold: 128  # The default value is 128.
 # sql-show is the same as props in logger ShardingSphere-SQL, and its priority is lower than logging rule
 sql-show: false
 check-table-metadata-enabled: false
   # Proxy backend query fetch size. A larger value may increase the memory usage of ShardingSphere Proxy.
   # The default value is -1, which means set the minimum value for different JDBC drivers.
 proxy-backend-query-fetch-size: -1
 proxy-frontend-executor-size: 0 # Proxy frontend executor size. The default value is 0, which means let Netty decide.
 proxy-frontend-max-connections: 0 # Less than or equal to 0 means no limitation.
 proxy-default-port: 3307 # Proxy default port.
 proxy-netty-backlog: 1024 # Proxy netty backlog.
 cdc-server-port: 33071 # CDC server port
 proxy-frontend-ssl-enabled: false
 proxy-frontend-ssl-cipher: ''
 proxy-frontend-ssl-version: TLSv1.2,TLSv1.3

配置参数可以去官网查询：Apache ShardingSphere

在config-readwrite-splitting.yaml进行读写分离以及分库分表相关配置

yaml 复制代码

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

######################################################################################################
#
# Here you can configure the rules for the proxy.
# This example is configuration of readwrite-splitting rule.
#
######################################################################################################
#
# 逻辑数据库名
databaseName: smart-education_db
# 数据源
dataSources:
  ds_0:
    url: jdbc:mysql://192.168.21.130:3306/tc_test_0?useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=true&serverTimezone=GMT%2B8&autoReconnect=true&rewriteBatchedStatements=true
    username: root
    password: 123456
    connectionTimeoutMilliseconds: 30000
    idleTimeoutMilliseconds: 60000
    maxLifetimeMilliseconds: 1800000
    maxPoolSize: 50
    minPoolSize: 1
  ds_read_0:
    url: jdbc:mysql://192.168.21.131:3306/tc_test_0?useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=true&serverTimezone=GMT%2B8&autoReconnect=true&rewriteBatchedStatements=true
    username: root
    password: 123456
    connectionTimeoutMilliseconds: 30000
    idleTimeoutMilliseconds: 60000
    maxLifetimeMilliseconds: 1800000
    maxPoolSize: 50
    minPoolSize: 1
  ds_1:
    url: jdbc:mysql://192.168.21.130:3306/tc_test_1?useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=true&serverTimezone=GMT%2B8&autoReconnect=true&rewriteBatchedStatements=true
    username: root
    password: 123456
    connectionTimeoutMilliseconds: 30000
    idleTimeoutMilliseconds: 60000
    maxLifetimeMilliseconds: 1800000
    maxPoolSize: 50
    minPoolSize: 1
  ds_read_1:
    url: jdbc:mysql://192.168.21.131:3306/tc_test_1?useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=true&serverTimezone=GMT%2B8&autoReconnect=true&rewriteBatchedStatements=true
    username: root
    password: 123456
    connectionTimeoutMilliseconds: 30000
    idleTimeoutMilliseconds: 60000
    maxLifetimeMilliseconds: 1800000
    maxPoolSize: 50
    minPoolSize: 1

rules:
- !READWRITE_SPLITTING
  dataSources:
    readwrite_ds_0:
      writeDataSourceName: ds_0
      readDataSourceNames:
        - ds_read_0
      loadBalancerName: random
    readwrite_ds_1:
      writeDataSourceName: ds_1
      readDataSourceNames:
        - ds_read_1
      loadBalancerName: random
  loadBalancers:
    random:
      type: RANDOM
# 单表
- !SINGLE
  tables:
    - readwrite_ds_0.*
  defaultDataSource: readwrite_ds_000000
  # 广播表
- !BROADCAST
  tables:
    
- !SHARDING
  tables: # 数据分片规则配置
    sys_user: # 文章审核逻辑表名称
      actualDataNodes: readwrite_ds_${0..1}.sys_user
      databaseStrategy: # 配置分库策略
        standard:
          shardingColumn: user_id
          shardingAlgorithmName: database_tenant_inline
      keyGenerateStrategy:
        column: id
        keyGeneratorName: snowflake

  # 分片算法配置
  shardingAlgorithms:
    database_tenant_inline:
      type: INLINE
      props:
        algorithm-expression: readwrite_ds_${user_id%2}

  # 分布式序列算法配置
  keyGenerators:
    snowflake:
      type: SNOWFLAKE
      props:
        worker-id: 1

重启sharding proxy

tex 复制代码

docker restart [容器id]

Navicat 客户端连接sharding proxy数据库端口3307 账户和密码都是root

测试数据插入和查询

sql 复制代码

insert sys_user (name) values ('张三');
insert sys_user (name) values ('李四');

至此搭建完成，感谢观赏。