MYSQL系列-分库分表（三）：Sharding-JDBC实现分库分表落地实践-下

技术驿站2023-09-24 22:26

系列文档参考 MYSQL系列-整体架构介绍

紧接上文MYSQL系列-分库分表（三）：Sharding-JDBC实现分库分表落地实践-中

详细设计

关键技术点实现

支持读写分离，并且某些不能有主从延迟的SQL强制走写库查询

sharding-jdbc本身支持读写分离，相关说明可以参考官网

yml文件配置如下

java 复制代码 
rules:
- !READWRITE_SPLITTING
  dataSources:
    readwrite_ds1:
      staticStrategy:
        writeDataSourceName: ds_1
        readDataSourceNames:
          - r_ds_1
      loadBalancerName: coupon_db_random
    readwrite_ds2:
      staticStrategy:
        writeDataSourceName: ds_2
        readDataSourceNames:
          - r_ds_2
      loadBalancerName: coupon_db_random
  loadBalancers:
    coupon_db_random:
      type: RANDOM

针对某些需要走写库的查询SQL，Sharding提供了HintManager强制走写库，示例如下

java 复制代码 
@Test
public void selectByCouponCode() {
    List<CouponInfo> couponInfoList = couponInfoMapper.selectByCouponCode("1234567ABCD", "CN");
    Assert.assertTrue(couponInfoList.size() > 0);
    LOGGER.info("couponInfoList={}", couponInfoList);
    HintManager.clear();
    //设置完需要移除
    HintManager.getInstance().setWriteRouteOnly();
    couponInfoList = couponInfoMapper.selectByCouponCode("1234567ABCD", "CN");
    Assert.assertTrue(couponInfoList.size() > 0);
    LOGGER.info("couponInfoList={}", couponInfoList);
    HintManager.clear();
}

支持影子库

sharding-jdbc也支持影子库特性，具体参考官网-影子库

如果使用其特性，配置起来会比较麻烦

实际影子库是为了支撑全链路压测的，为了和现网实际业务隔离开，应该把所有的表都放在新的数据库即影子库上面

采用上下文带影子库标记，如果有则在路由算法地方将数据库偏移到对应的影子库

具体代码如下：

首先提供下上文处理类

java 复制代码 
public class Context {
    private static final ThreadLocal<Context> CONTEXT = new ThreadLocal<Context>() {
        @Override
        protected Context initialValue() {
            return new Context();
        }
    };

    /**
     * 是否影子库标记常量
     */
    private static final String SHADOW = "YC";

    /**
     * 影子库标记
     */
    private static final String IS_SHADOW = "1";
    
    private final Map<String, String> parameters = new HashMap<>();
    private final Map<String, String> localValues = new HashMap<>();
 
    ...
    public static void clear() {
        CONTEXT.remove();
    }

    public static void setShadow() {
        CONTEXT.get().parameters.put(SHADOW, IS_SHADOW);
    }

    public static boolean isShadow() {
        return IS_SHADOW.equals(CONTEXT.get().parameters.get(SHADOW));
    }

}

yml文件多配置一倍数据库实例作为影子库

yaml 复制代码 
dataSources:
  ds_1:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
  ds_2:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
  ds_3:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  ds_4:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0

- !SHARDING
  tables:
    point_balance:
      actualDataNodes: ds_${1..4}.point_balance${1..2}
      databaseStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: d_uid_inline
      tableStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: t_uid_inline
      auditStrategy:
        auditorNames:
          - sharding_key_required_auditor
        allowHintDisable: true

CodeDbSharding和UIDDbSharding获取分库索引时加上偏移量

java 复制代码 
@Override
protected long getMode(long mode) {
    if (Context.isShadow()) {
        mode += 2;
    }
    return mode;
}

实现效果类似如下

java 复制代码 
@Test
public void testShadow() {
    Context.setShadow();
    List<CouponInfo> couponInfoList = couponInfoMapper.selectByCouponCode("1234567ABCD", "CN");
    Assert.assertTrue(couponInfoList.size() > 0);
    Context.clear();
    couponInfoList = couponInfoMapper.selectByCouponCode("1234567ABCD", "CN");
    Assert.assertTrue(couponInfoList.size() > 0);
}

执行结果如下,第一次带上标记落在ds_4,没带上影子库标记落在ds_2

yaml 复制代码 
2023-09-24 22:09:05.101|INFO |demo|1|127.0.0.1|1091ceaed1fe445a83a7265a6ef84065|Actual SQL: ds_4 ::: select id,         coupon_code,         rev_uid,         create_time,         modify_time,         country        from coupon_info1 where coupon_code=? and country=? ::: [1234567ABCD, CN]|ShardingSphere-SQL
<==    Columns: id, coupon_code, rev_uid, create_time, modify_time, country
<==        Row: 1, 1234567ABCD, 3, 2023-09-24 17:21:59.0, 2023-09-24 17:21:59.0, CN
<==        Row: 2, 1234567ABCD, 3, 2023-09-24 17:22:13.0, 2023-09-24 17:22:13.0, CN
<==        Row: 3, 1234567ABCD, 3, 2023-09-24 17:24:41.0, 2023-09-24 17:24:41.0, CN
<==        Row: 4, 1234567ABCD, 3, 2023-09-24 17:24:41.0, 2023-09-24 17:24:41.0, CN
<==      Total: 4
Closing non transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@79bf39e5]
Creating a new SqlSession
SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@354baab2] was not registered for synchronization because synchronization is not active
2023-09-24 22:09:05.184|DEBUG|demo|1|127.0.0.1|1091ceaed1fe445a83a7265a6ef84065|Fetching JDBC Connection from DataSource|org.springframework.jdbc.datasource.DataSourceUtils
JDBC Connection [org.apache.shardingsphere.driver.jdbc.core.connection.ShardingSphereConnection@30be6a05] will not be managed by Spring
==>  Preparing: select id, coupon_code, rev_uid, create_time, modify_time, country from coupon_info where coupon_code=? and country=? 
==> Parameters: 1234567ABCD(String), CN(String)
2023-09-24 22:09:05.184|INFO |demo|1|127.0.0.1|1091ceaed1fe445a83a7265a6ef84065|Logic SQL: select id,         coupon_code,         rev_uid,         create_time,         modify_time,         country        from coupon_info where coupon_code=? and country=?|ShardingSphere-SQL
2023-09-24 22:09:05.184|INFO |demo|1|127.0.0.1|1091ceaed1fe445a83a7265a6ef84065|Actual SQL: ds_2 ::: select id,         coupon_code,         rev_uid,         create_time,         modify_time,         country        from coupon_info1 where coupon_code=? and country=? ::: [1234567ABCD, CN]|ShardingSphere-SQL
<==    Columns: id, coupon_code, rev_uid, create_time, modify_time, country
<==        Row: 1, 1234567ABCD, 3, 2023-09-24 17:21:59.0, 2023-09-24 17:21:59.0, CN
<==        Row: 2, 1234567ABCD, 3, 2023-09-24 17:22:13.0, 2023-09-24 17:22:13.0, CN
<==        Row: 3, 1234567ABCD, 3, 2023-09-24 17:24:41.0, 2023-09-24 17:24:41.0, CN
<==        Row: 4, 1234567ABCD, 3, 2023-09-24 17:24:41.0, 2023-09-24 17:24:41.0, CN
<==      Total: 4
Closing non transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@354baab2]

针对海外某些国家，数据量较少，业务表也不进行分库分表处理

这块当前没有实现，需要在分库的路由算法增加国家维度，不同国家落到不同的库。这样使用yml配置不太便捷，需要用java代码来创建datasource

最终yml配置文件和整体代码结构

sharding-databases-tables.yaml文件如下

yaml 复制代码 
mode:
  type: Standalone
  repository:
    type: JDBC
dataSources:
  ds_1:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
  ds_2:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
  r_ds_1:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  r_ds_2:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  ds_3:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  ds_4:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  r_ds_3:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  r_ds_4:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
rules:
- !READWRITE_SPLITTING
  dataSources:
    readwrite_ds1:
      staticStrategy:
        writeDataSourceName: ds_1
        readDataSourceNames:
          - r_ds_1
      loadBalancerName: coupon_db_random
    readwrite_ds2:
      staticStrategy:
        writeDataSourceName: ds_2
        readDataSourceNames:
          - r_ds_2
      loadBalancerName: coupon_db_random
  loadBalancers:
    coupon_db_random:
      type: RANDOM

- !SHARDING
  tables:
    point_balance:
      actualDataNodes: ds_${1..4}.point_balance${1..2}
      databaseStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: d_uid_inline
      tableStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: t_uid_inline
      auditStrategy:
        auditorNames:
          - sharding_key_required_auditor
        allowHintDisable: true
    point_balance_his:
      actualDataNodes: ds_${1..4}.point_balance_his${1..2}
      databaseStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: d_uid_inline
      tableStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: t_uid_inline
      auditStrategy:
        auditorNames:
          - sharding_key_required_auditor
        allowHintDisable: true
    param_config:
      actualDataNodes: ds_1.param_config
      databaseStrategy:
        none:
      tableStrategy:
        none:
    coupon_info:
      actualDataNodes: ds_${1..4}.coupon_info${1..2}
      databaseStrategy:
        standard:
          shardingColumn: coupon_code
          shardingAlgorithmName: d_code_inline
      tableStrategy:
        standard:
          shardingColumn: coupon_code
          shardingAlgorithmName: t_code_inline
      auditStrategy:
        auditorNames:
          - sharding_key_required_auditor
        allowHintDisable: true
  defaultShardingColumn: uid
  bindingTables:
    - point_balance,point_balance_his
    - coupon_info
  defaultDatabaseStrategy:
    none:
  defaultTableStrategy:
    none:
  shardingAlgorithms:
    d_uid_inline:
      type: CLASS_BASED
      props:
        algorithmClassName: com.toby.sharding.jdbc.source.start.db.sharding.UIDDbSharding
        strategy: STANDARD
    t_uid_inline:
      type: CLASS_BASED
      props:
        algorithmClassName: com.toby.sharding.jdbc.source.start.db.sharding.UIDTableSharding
        strategy: STANDARD
    d_code_inline:
      type: CLASS_BASED
      props:
        algorithmClassName: com.toby.sharding.jdbc.source.start.db.sharding.CodeDbSharding
        strategy: STANDARD
    t_code_inline:
      type: CLASS_BASED
      props:
        algorithmClassName: com.toby.sharding.jdbc.source.start.db.sharding.CodeTableSharding
        strategy: STANDARD
  auditors:
    sharding_key_required_auditor:
      type: DML_SHARDING_CONDITIONS
#  - !BROADCAST
#    tables:
#      -
props:
  sql-show: true

项目文件结构如下

项目代码在git目录，github.com/hongyuwen/H...

相关推荐
㳺三才人子3 小时前
初探 Flask
后端·python·flask·html
星栈独行3 小时前
我在 Rust 全栈项目里用 JWT 做无状态认证
开发语言·后端·rust·前端框架·开源·github·web
Java爱好狂.4 小时前
Java程序员体系化学习路线（2026最新版）
java·后端·java面试·java架构师·java程序员·java八股文·java学习路线
陈随易4 小时前
Redis 8.8发布，一定要更新
前端·后端·程序员
装不满的克莱因瓶4 小时前
SpringBoot 如何将 lib 目录中jar包打包进最终的jar包里面
spring boot·后端·maven·jar·mvn
Raink老师5 小时前
【AI面试临阵磨枪-62】设计基于 RAG 的内部知识库问答平台（多租户、权限、文件上传、实时更新）
人工智能·面试·职场和发展
IronMurphy5 小时前
MySQL拷打第二讲
数据库·mysql
ltl5 小时前
Transformer 原论文实验结果：为什么 28.4 BLEU 足以改写路线图
后端
excel6 小时前
为什么我推荐使用 Termius：现代 SSH 工具的完整体验
前端·后端
卷毛的技术笔记6 小时前
Java后端硬核实战：用Spring AI Alibaba+Redis给LLM装上“超强记忆中枢”
java·人工智能·redis·后端·spring·ai·系统架构
热门推荐
01GitHub 镜像站点02Codex 接入 DeepSeek API 完整配置文档03【踩坑记录 | 第一篇】微软商店无法使用时，如何手动安装 OpenAI Codex？附`.msix`文件系统错误解决方法04裂开！ChatGPT 居然开始要手机号验证，附详细解决方法05CC-Switch & Claude 基于 Linux 服务器安装使用指南06【AI】2026 年具身智能模型和世界模型总结07Codegraph 实战：用知识图谱让 AI 编程效率翻倍08LLM Wiki：让大模型替你打理知识库的完整指南09Codex使用DeepSeek API的方法（cc switch + codex bridge方案）10几个好用的ip纯净度检测网站