MYSQL系列-分库分表（三）：Sharding-JDBC实现分库分表落地实践-下

技术驿站2023-09-24 22:26

系列文档参考 MYSQL系列-整体架构介绍

紧接上文MYSQL系列-分库分表（三）：Sharding-JDBC实现分库分表落地实践-中

详细设计

关键技术点实现

支持读写分离，并且某些不能有主从延迟的SQL强制走写库查询

sharding-jdbc本身支持读写分离，相关说明可以参考官网

yml文件配置如下

java 复制代码 
rules:
- !READWRITE_SPLITTING
  dataSources:
    readwrite_ds1:
      staticStrategy:
        writeDataSourceName: ds_1
        readDataSourceNames:
          - r_ds_1
      loadBalancerName: coupon_db_random
    readwrite_ds2:
      staticStrategy:
        writeDataSourceName: ds_2
        readDataSourceNames:
          - r_ds_2
      loadBalancerName: coupon_db_random
  loadBalancers:
    coupon_db_random:
      type: RANDOM

针对某些需要走写库的查询SQL，Sharding提供了HintManager强制走写库，示例如下

java 复制代码 
@Test
public void selectByCouponCode() {
    List<CouponInfo> couponInfoList = couponInfoMapper.selectByCouponCode("1234567ABCD", "CN");
    Assert.assertTrue(couponInfoList.size() > 0);
    LOGGER.info("couponInfoList={}", couponInfoList);
    HintManager.clear();
    //设置完需要移除
    HintManager.getInstance().setWriteRouteOnly();
    couponInfoList = couponInfoMapper.selectByCouponCode("1234567ABCD", "CN");
    Assert.assertTrue(couponInfoList.size() > 0);
    LOGGER.info("couponInfoList={}", couponInfoList);
    HintManager.clear();
}

支持影子库

sharding-jdbc也支持影子库特性，具体参考官网-影子库

如果使用其特性，配置起来会比较麻烦

实际影子库是为了支撑全链路压测的，为了和现网实际业务隔离开，应该把所有的表都放在新的数据库即影子库上面

采用上下文带影子库标记，如果有则在路由算法地方将数据库偏移到对应的影子库

具体代码如下：

首先提供下上文处理类

java 复制代码 
public class Context {
    private static final ThreadLocal<Context> CONTEXT = new ThreadLocal<Context>() {
        @Override
        protected Context initialValue() {
            return new Context();
        }
    };

    /**
     * 是否影子库标记常量
     */
    private static final String SHADOW = "YC";

    /**
     * 影子库标记
     */
    private static final String IS_SHADOW = "1";
    
    private final Map<String, String> parameters = new HashMap<>();
    private final Map<String, String> localValues = new HashMap<>();
 
    ...
    public static void clear() {
        CONTEXT.remove();
    }

    public static void setShadow() {
        CONTEXT.get().parameters.put(SHADOW, IS_SHADOW);
    }

    public static boolean isShadow() {
        return IS_SHADOW.equals(CONTEXT.get().parameters.get(SHADOW));
    }

}

yml文件多配置一倍数据库实例作为影子库

yaml 复制代码 
dataSources:
  ds_1:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
  ds_2:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
  ds_3:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  ds_4:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0

- !SHARDING
  tables:
    point_balance:
      actualDataNodes: ds_${1..4}.point_balance${1..2}
      databaseStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: d_uid_inline
      tableStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: t_uid_inline
      auditStrategy:
        auditorNames:
          - sharding_key_required_auditor
        allowHintDisable: true

CodeDbSharding和UIDDbSharding获取分库索引时加上偏移量

java 复制代码 
@Override
protected long getMode(long mode) {
    if (Context.isShadow()) {
        mode += 2;
    }
    return mode;
}

实现效果类似如下

java 复制代码 
@Test
public void testShadow() {
    Context.setShadow();
    List<CouponInfo> couponInfoList = couponInfoMapper.selectByCouponCode("1234567ABCD", "CN");
    Assert.assertTrue(couponInfoList.size() > 0);
    Context.clear();
    couponInfoList = couponInfoMapper.selectByCouponCode("1234567ABCD", "CN");
    Assert.assertTrue(couponInfoList.size() > 0);
}

执行结果如下,第一次带上标记落在ds_4,没带上影子库标记落在ds_2

yaml 复制代码 
2023-09-24 22:09:05.101|INFO |demo|1|127.0.0.1|1091ceaed1fe445a83a7265a6ef84065|Actual SQL: ds_4 ::: select id,         coupon_code,         rev_uid,         create_time,         modify_time,         country        from coupon_info1 where coupon_code=? and country=? ::: [1234567ABCD, CN]|ShardingSphere-SQL
<==    Columns: id, coupon_code, rev_uid, create_time, modify_time, country
<==        Row: 1, 1234567ABCD, 3, 2023-09-24 17:21:59.0, 2023-09-24 17:21:59.0, CN
<==        Row: 2, 1234567ABCD, 3, 2023-09-24 17:22:13.0, 2023-09-24 17:22:13.0, CN
<==        Row: 3, 1234567ABCD, 3, 2023-09-24 17:24:41.0, 2023-09-24 17:24:41.0, CN
<==        Row: 4, 1234567ABCD, 3, 2023-09-24 17:24:41.0, 2023-09-24 17:24:41.0, CN
<==      Total: 4
Closing non transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@79bf39e5]
Creating a new SqlSession
SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@354baab2] was not registered for synchronization because synchronization is not active
2023-09-24 22:09:05.184|DEBUG|demo|1|127.0.0.1|1091ceaed1fe445a83a7265a6ef84065|Fetching JDBC Connection from DataSource|org.springframework.jdbc.datasource.DataSourceUtils
JDBC Connection [org.apache.shardingsphere.driver.jdbc.core.connection.ShardingSphereConnection@30be6a05] will not be managed by Spring
==>  Preparing: select id, coupon_code, rev_uid, create_time, modify_time, country from coupon_info where coupon_code=? and country=? 
==> Parameters: 1234567ABCD(String), CN(String)
2023-09-24 22:09:05.184|INFO |demo|1|127.0.0.1|1091ceaed1fe445a83a7265a6ef84065|Logic SQL: select id,         coupon_code,         rev_uid,         create_time,         modify_time,         country        from coupon_info where coupon_code=? and country=?|ShardingSphere-SQL
2023-09-24 22:09:05.184|INFO |demo|1|127.0.0.1|1091ceaed1fe445a83a7265a6ef84065|Actual SQL: ds_2 ::: select id,         coupon_code,         rev_uid,         create_time,         modify_time,         country        from coupon_info1 where coupon_code=? and country=? ::: [1234567ABCD, CN]|ShardingSphere-SQL
<==    Columns: id, coupon_code, rev_uid, create_time, modify_time, country
<==        Row: 1, 1234567ABCD, 3, 2023-09-24 17:21:59.0, 2023-09-24 17:21:59.0, CN
<==        Row: 2, 1234567ABCD, 3, 2023-09-24 17:22:13.0, 2023-09-24 17:22:13.0, CN
<==        Row: 3, 1234567ABCD, 3, 2023-09-24 17:24:41.0, 2023-09-24 17:24:41.0, CN
<==        Row: 4, 1234567ABCD, 3, 2023-09-24 17:24:41.0, 2023-09-24 17:24:41.0, CN
<==      Total: 4
Closing non transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@354baab2]

针对海外某些国家，数据量较少，业务表也不进行分库分表处理

这块当前没有实现，需要在分库的路由算法增加国家维度，不同国家落到不同的库。这样使用yml配置不太便捷，需要用java代码来创建datasource

最终yml配置文件和整体代码结构

sharding-databases-tables.yaml文件如下

yaml 复制代码 
mode:
  type: Standalone
  repository:
    type: JDBC
dataSources:
  ds_1:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
  ds_2:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
  r_ds_1:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  r_ds_2:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  ds_3:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  ds_4:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  r_ds_3:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  r_ds_4:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
rules:
- !READWRITE_SPLITTING
  dataSources:
    readwrite_ds1:
      staticStrategy:
        writeDataSourceName: ds_1
        readDataSourceNames:
          - r_ds_1
      loadBalancerName: coupon_db_random
    readwrite_ds2:
      staticStrategy:
        writeDataSourceName: ds_2
        readDataSourceNames:
          - r_ds_2
      loadBalancerName: coupon_db_random
  loadBalancers:
    coupon_db_random:
      type: RANDOM

- !SHARDING
  tables:
    point_balance:
      actualDataNodes: ds_${1..4}.point_balance${1..2}
      databaseStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: d_uid_inline
      tableStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: t_uid_inline
      auditStrategy:
        auditorNames:
          - sharding_key_required_auditor
        allowHintDisable: true
    point_balance_his:
      actualDataNodes: ds_${1..4}.point_balance_his${1..2}
      databaseStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: d_uid_inline
      tableStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: t_uid_inline
      auditStrategy:
        auditorNames:
          - sharding_key_required_auditor
        allowHintDisable: true
    param_config:
      actualDataNodes: ds_1.param_config
      databaseStrategy:
        none:
      tableStrategy:
        none:
    coupon_info:
      actualDataNodes: ds_${1..4}.coupon_info${1..2}
      databaseStrategy:
        standard:
          shardingColumn: coupon_code
          shardingAlgorithmName: d_code_inline
      tableStrategy:
        standard:
          shardingColumn: coupon_code
          shardingAlgorithmName: t_code_inline
      auditStrategy:
        auditorNames:
          - sharding_key_required_auditor
        allowHintDisable: true
  defaultShardingColumn: uid
  bindingTables:
    - point_balance,point_balance_his
    - coupon_info
  defaultDatabaseStrategy:
    none:
  defaultTableStrategy:
    none:
  shardingAlgorithms:
    d_uid_inline:
      type: CLASS_BASED
      props:
        algorithmClassName: com.toby.sharding.jdbc.source.start.db.sharding.UIDDbSharding
        strategy: STANDARD
    t_uid_inline:
      type: CLASS_BASED
      props:
        algorithmClassName: com.toby.sharding.jdbc.source.start.db.sharding.UIDTableSharding
        strategy: STANDARD
    d_code_inline:
      type: CLASS_BASED
      props:
        algorithmClassName: com.toby.sharding.jdbc.source.start.db.sharding.CodeDbSharding
        strategy: STANDARD
    t_code_inline:
      type: CLASS_BASED
      props:
        algorithmClassName: com.toby.sharding.jdbc.source.start.db.sharding.CodeTableSharding
        strategy: STANDARD
  auditors:
    sharding_key_required_auditor:
      type: DML_SHARDING_CONDITIONS
#  - !BROADCAST
#    tables:
#      -
props:
  sql-show: true

项目文件结构如下

项目代码在git目录，github.com/hongyuwen/H...

相关推荐
monkey_meng15 分钟前
【Rust中的迭代器】
开发语言·后端·rust
余衫马18 分钟前
Rust-Trait 特征编程
开发语言·后端·rust
monkey_meng22 分钟前
【Rust中多线程同步机制】
开发语言·redis·后端·rust
阑梦清川1 小时前
在鱼皮的模拟面试里面学习有感
学习·面试·职场和发展
瓜牛_gn2 小时前
mysql特性
数据库·mysql
paopaokaka_luck5 小时前
【360】基于springboot的志愿服务管理系统
java·spring boot·后端·spring·毕业设计
码农小旋风6 小时前
详解K8S--声明式API
后端
Peter_chq6 小时前
【操作系统】基于环形队列的生产消费模型
linux·c语言·开发语言·c++·后端
Yaml47 小时前
Spring Boot 与 Vue 共筑二手书籍交易卓越平台
java·spring boot·后端·mysql·spring·vue·二手书籍
小小小妮子~7 小时前
Spring Boot详解：从入门到精通
java·spring boot·后端
热门推荐
01【HarmonyOS】HUAWEI DevEco Studio 下载地址汇总02(欧拉)openEuler系统添加网卡文件配置流程、(欧拉)openEuler系统手动配置ipv6地址流程、(欧拉)openEuler系统网络管理说明03组基轨迹建模 GBTM的介绍与实现（Stata 或 R）04【AIGC】重塑未来的科技巨轮05全面解析：构建基于深度学习的安全帽检测系统（UI界面+YOLO代码+数据集）06【经验分享】Ubuntu22.04安装微信(linux官方版)07基于YOLOv10深度学习的CT扫描图像肾结石智能检测系统【python源码+Pyqt5界面+数据集+训练代码】深度学习实战、目标检测08Ubuntu 20.04使用Livox mid 360 测试 FAST_LIO09RAG 实践- Ollama+RagFlow 部署本地知识库10红米手机使用google play