MYSQL系列-分库分表（三）：Sharding-JDBC实现分库分表落地实践-下

技术驿站2023-09-24 22:26

系列文档参考 MYSQL系列-整体架构介绍

紧接上文MYSQL系列-分库分表（三）：Sharding-JDBC实现分库分表落地实践-中

详细设计

关键技术点实现

支持读写分离，并且某些不能有主从延迟的SQL强制走写库查询

sharding-jdbc本身支持读写分离，相关说明可以参考官网

yml文件配置如下

java 复制代码 
rules:
- !READWRITE_SPLITTING
  dataSources:
    readwrite_ds1:
      staticStrategy:
        writeDataSourceName: ds_1
        readDataSourceNames:
          - r_ds_1
      loadBalancerName: coupon_db_random
    readwrite_ds2:
      staticStrategy:
        writeDataSourceName: ds_2
        readDataSourceNames:
          - r_ds_2
      loadBalancerName: coupon_db_random
  loadBalancers:
    coupon_db_random:
      type: RANDOM

针对某些需要走写库的查询SQL，Sharding提供了HintManager强制走写库，示例如下

java 复制代码 
@Test
public void selectByCouponCode() {
    List<CouponInfo> couponInfoList = couponInfoMapper.selectByCouponCode("1234567ABCD", "CN");
    Assert.assertTrue(couponInfoList.size() > 0);
    LOGGER.info("couponInfoList={}", couponInfoList);
    HintManager.clear();
    //设置完需要移除
    HintManager.getInstance().setWriteRouteOnly();
    couponInfoList = couponInfoMapper.selectByCouponCode("1234567ABCD", "CN");
    Assert.assertTrue(couponInfoList.size() > 0);
    LOGGER.info("couponInfoList={}", couponInfoList);
    HintManager.clear();
}

支持影子库

sharding-jdbc也支持影子库特性，具体参考官网-影子库

如果使用其特性，配置起来会比较麻烦

实际影子库是为了支撑全链路压测的，为了和现网实际业务隔离开，应该把所有的表都放在新的数据库即影子库上面

采用上下文带影子库标记，如果有则在路由算法地方将数据库偏移到对应的影子库

具体代码如下：

首先提供下上文处理类

java 复制代码 
public class Context {
    private static final ThreadLocal<Context> CONTEXT = new ThreadLocal<Context>() {
        @Override
        protected Context initialValue() {
            return new Context();
        }
    };

    /**
     * 是否影子库标记常量
     */
    private static final String SHADOW = "YC";

    /**
     * 影子库标记
     */
    private static final String IS_SHADOW = "1";
    
    private final Map<String, String> parameters = new HashMap<>();
    private final Map<String, String> localValues = new HashMap<>();
 
    ...
    public static void clear() {
        CONTEXT.remove();
    }

    public static void setShadow() {
        CONTEXT.get().parameters.put(SHADOW, IS_SHADOW);
    }

    public static boolean isShadow() {
        return IS_SHADOW.equals(CONTEXT.get().parameters.get(SHADOW));
    }

}

yml文件多配置一倍数据库实例作为影子库

yaml 复制代码 
dataSources:
  ds_1:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
  ds_2:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
  ds_3:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  ds_4:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0

- !SHARDING
  tables:
    point_balance:
      actualDataNodes: ds_${1..4}.point_balance${1..2}
      databaseStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: d_uid_inline
      tableStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: t_uid_inline
      auditStrategy:
        auditorNames:
          - sharding_key_required_auditor
        allowHintDisable: true

CodeDbSharding和UIDDbSharding获取分库索引时加上偏移量

java 复制代码 
@Override
protected long getMode(long mode) {
    if (Context.isShadow()) {
        mode += 2;
    }
    return mode;
}

实现效果类似如下

java 复制代码 
@Test
public void testShadow() {
    Context.setShadow();
    List<CouponInfo> couponInfoList = couponInfoMapper.selectByCouponCode("1234567ABCD", "CN");
    Assert.assertTrue(couponInfoList.size() > 0);
    Context.clear();
    couponInfoList = couponInfoMapper.selectByCouponCode("1234567ABCD", "CN");
    Assert.assertTrue(couponInfoList.size() > 0);
}

执行结果如下,第一次带上标记落在ds_4,没带上影子库标记落在ds_2

yaml 复制代码 
2023-09-24 22:09:05.101|INFO |demo|1|127.0.0.1|1091ceaed1fe445a83a7265a6ef84065|Actual SQL: ds_4 ::: select id,         coupon_code,         rev_uid,         create_time,         modify_time,         country        from coupon_info1 where coupon_code=? and country=? ::: [1234567ABCD, CN]|ShardingSphere-SQL
<==    Columns: id, coupon_code, rev_uid, create_time, modify_time, country
<==        Row: 1, 1234567ABCD, 3, 2023-09-24 17:21:59.0, 2023-09-24 17:21:59.0, CN
<==        Row: 2, 1234567ABCD, 3, 2023-09-24 17:22:13.0, 2023-09-24 17:22:13.0, CN
<==        Row: 3, 1234567ABCD, 3, 2023-09-24 17:24:41.0, 2023-09-24 17:24:41.0, CN
<==        Row: 4, 1234567ABCD, 3, 2023-09-24 17:24:41.0, 2023-09-24 17:24:41.0, CN
<==      Total: 4
Closing non transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@79bf39e5]
Creating a new SqlSession
SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@354baab2] was not registered for synchronization because synchronization is not active
2023-09-24 22:09:05.184|DEBUG|demo|1|127.0.0.1|1091ceaed1fe445a83a7265a6ef84065|Fetching JDBC Connection from DataSource|org.springframework.jdbc.datasource.DataSourceUtils
JDBC Connection [org.apache.shardingsphere.driver.jdbc.core.connection.ShardingSphereConnection@30be6a05] will not be managed by Spring
==>  Preparing: select id, coupon_code, rev_uid, create_time, modify_time, country from coupon_info where coupon_code=? and country=? 
==> Parameters: 1234567ABCD(String), CN(String)
2023-09-24 22:09:05.184|INFO |demo|1|127.0.0.1|1091ceaed1fe445a83a7265a6ef84065|Logic SQL: select id,         coupon_code,         rev_uid,         create_time,         modify_time,         country        from coupon_info where coupon_code=? and country=?|ShardingSphere-SQL
2023-09-24 22:09:05.184|INFO |demo|1|127.0.0.1|1091ceaed1fe445a83a7265a6ef84065|Actual SQL: ds_2 ::: select id,         coupon_code,         rev_uid,         create_time,         modify_time,         country        from coupon_info1 where coupon_code=? and country=? ::: [1234567ABCD, CN]|ShardingSphere-SQL
<==    Columns: id, coupon_code, rev_uid, create_time, modify_time, country
<==        Row: 1, 1234567ABCD, 3, 2023-09-24 17:21:59.0, 2023-09-24 17:21:59.0, CN
<==        Row: 2, 1234567ABCD, 3, 2023-09-24 17:22:13.0, 2023-09-24 17:22:13.0, CN
<==        Row: 3, 1234567ABCD, 3, 2023-09-24 17:24:41.0, 2023-09-24 17:24:41.0, CN
<==        Row: 4, 1234567ABCD, 3, 2023-09-24 17:24:41.0, 2023-09-24 17:24:41.0, CN
<==      Total: 4
Closing non transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@354baab2]

针对海外某些国家，数据量较少，业务表也不进行分库分表处理

这块当前没有实现，需要在分库的路由算法增加国家维度，不同国家落到不同的库。这样使用yml配置不太便捷，需要用java代码来创建datasource

最终yml配置文件和整体代码结构

sharding-databases-tables.yaml文件如下

yaml 复制代码 
mode:
  type: Standalone
  repository:
    type: JDBC
dataSources:
  ds_1:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
  ds_2:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
  r_ds_1:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  r_ds_2:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  ds_3:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  ds_4:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  r_ds_3:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  r_ds_4:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
rules:
- !READWRITE_SPLITTING
  dataSources:
    readwrite_ds1:
      staticStrategy:
        writeDataSourceName: ds_1
        readDataSourceNames:
          - r_ds_1
      loadBalancerName: coupon_db_random
    readwrite_ds2:
      staticStrategy:
        writeDataSourceName: ds_2
        readDataSourceNames:
          - r_ds_2
      loadBalancerName: coupon_db_random
  loadBalancers:
    coupon_db_random:
      type: RANDOM

- !SHARDING
  tables:
    point_balance:
      actualDataNodes: ds_${1..4}.point_balance${1..2}
      databaseStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: d_uid_inline
      tableStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: t_uid_inline
      auditStrategy:
        auditorNames:
          - sharding_key_required_auditor
        allowHintDisable: true
    point_balance_his:
      actualDataNodes: ds_${1..4}.point_balance_his${1..2}
      databaseStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: d_uid_inline
      tableStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: t_uid_inline
      auditStrategy:
        auditorNames:
          - sharding_key_required_auditor
        allowHintDisable: true
    param_config:
      actualDataNodes: ds_1.param_config
      databaseStrategy:
        none:
      tableStrategy:
        none:
    coupon_info:
      actualDataNodes: ds_${1..4}.coupon_info${1..2}
      databaseStrategy:
        standard:
          shardingColumn: coupon_code
          shardingAlgorithmName: d_code_inline
      tableStrategy:
        standard:
          shardingColumn: coupon_code
          shardingAlgorithmName: t_code_inline
      auditStrategy:
        auditorNames:
          - sharding_key_required_auditor
        allowHintDisable: true
  defaultShardingColumn: uid
  bindingTables:
    - point_balance,point_balance_his
    - coupon_info
  defaultDatabaseStrategy:
    none:
  defaultTableStrategy:
    none:
  shardingAlgorithms:
    d_uid_inline:
      type: CLASS_BASED
      props:
        algorithmClassName: com.toby.sharding.jdbc.source.start.db.sharding.UIDDbSharding
        strategy: STANDARD
    t_uid_inline:
      type: CLASS_BASED
      props:
        algorithmClassName: com.toby.sharding.jdbc.source.start.db.sharding.UIDTableSharding
        strategy: STANDARD
    d_code_inline:
      type: CLASS_BASED
      props:
        algorithmClassName: com.toby.sharding.jdbc.source.start.db.sharding.CodeDbSharding
        strategy: STANDARD
    t_code_inline:
      type: CLASS_BASED
      props:
        algorithmClassName: com.toby.sharding.jdbc.source.start.db.sharding.CodeTableSharding
        strategy: STANDARD
  auditors:
    sharding_key_required_auditor:
      type: DML_SHARDING_CONDITIONS
#  - !BROADCAST
#    tables:
#      -
props:
  sql-show: true

项目文件结构如下

项目代码在git目录，github.com/hongyuwen/H...

相关推荐
测试界柠檬11 分钟前
面试真题 | web自动化关闭浏览器，quit()和close()的区别
前端·自动化测试·软件测试·功能测试·程序人生·面试·自动化
hai4058714 分钟前
Spring Boot中的响应与分层解耦架构
spring boot·后端·架构
阿华的代码王国44 分钟前
MySQL ------- 索引（B树B+树）
数据库·mysql
Redstone Monstrosity1 小时前
字节二面
前端·面试
liupenglove1 小时前
golang操作mysql利器-gorm
mysql·golang
Adolf_19931 小时前
Flask-JWT-Extended登录验证, 不用自定义
后端·python·flask
叫我：松哥2 小时前
基于Python flask的医院管理学院，医生能够增加/删除/修改/删除病人的数据信息，有可视化分析
javascript·后端·python·mysql·信息可视化·flask·bootstrap
海里真的有鱼2 小时前
Spring Boot 项目中整合 RabbitMQ，使用死信队列（Dead Letter Exchange, DLX）实现延迟队列功能
开发语言·后端·rabbitmq
UestcXiye2 小时前
面试算法题精讲：求数组两组数差值和的最大值
面试·数据结构与算法·前后缀分解
严格格2 小时前
三范式，面试重点
数据库·面试·职场和发展
热门推荐
01RAG 实践- Ollama+RagFlow 部署本地知识库02组基轨迹建模 GBTM的介绍与实现（Stata 或 R）032024年高教社杯数学建模国赛C题超详细解题思路分析04苍穹外卖面试总结05yolov8实战第五天——yolov8+ffmpg实时视频流检测并进行实时推流——（推流，保姆教学）06【2024数模国赛赛题思路公开】国赛B题思路丨附可运行代码丨无偿自提07Coze扣子平台完整体验和实践（附国内和国际版对比）08CCF-CSP认证考试 202406-3 文本分词 100分题解09【2024高教社杯全国大学生数学建模竞赛】B题 生产过程中的决策问题——解题思路 代码 论文10借助 LocatorJS ，快速定位本地代码