MYSQL系列-分库分表（三）：Sharding-JDBC实现分库分表落地实践-下

技术驿站2023-09-24 22:26

系列文档参考 MYSQL系列-整体架构介绍

紧接上文MYSQL系列-分库分表（三）：Sharding-JDBC实现分库分表落地实践-中

详细设计

关键技术点实现

支持读写分离，并且某些不能有主从延迟的SQL强制走写库查询

sharding-jdbc本身支持读写分离，相关说明可以参考官网

yml文件配置如下

java 复制代码 
rules:
- !READWRITE_SPLITTING
  dataSources:
    readwrite_ds1:
      staticStrategy:
        writeDataSourceName: ds_1
        readDataSourceNames:
          - r_ds_1
      loadBalancerName: coupon_db_random
    readwrite_ds2:
      staticStrategy:
        writeDataSourceName: ds_2
        readDataSourceNames:
          - r_ds_2
      loadBalancerName: coupon_db_random
  loadBalancers:
    coupon_db_random:
      type: RANDOM

针对某些需要走写库的查询SQL，Sharding提供了HintManager强制走写库，示例如下

java 复制代码 
@Test
public void selectByCouponCode() {
    List<CouponInfo> couponInfoList = couponInfoMapper.selectByCouponCode("1234567ABCD", "CN");
    Assert.assertTrue(couponInfoList.size() > 0);
    LOGGER.info("couponInfoList={}", couponInfoList);
    HintManager.clear();
    //设置完需要移除
    HintManager.getInstance().setWriteRouteOnly();
    couponInfoList = couponInfoMapper.selectByCouponCode("1234567ABCD", "CN");
    Assert.assertTrue(couponInfoList.size() > 0);
    LOGGER.info("couponInfoList={}", couponInfoList);
    HintManager.clear();
}

支持影子库

sharding-jdbc也支持影子库特性，具体参考官网-影子库

如果使用其特性，配置起来会比较麻烦

实际影子库是为了支撑全链路压测的，为了和现网实际业务隔离开，应该把所有的表都放在新的数据库即影子库上面

采用上下文带影子库标记，如果有则在路由算法地方将数据库偏移到对应的影子库

具体代码如下：

首先提供下上文处理类

java 复制代码 
public class Context {
    private static final ThreadLocal<Context> CONTEXT = new ThreadLocal<Context>() {
        @Override
        protected Context initialValue() {
            return new Context();
        }
    };

    /**
     * 是否影子库标记常量
     */
    private static final String SHADOW = "YC";

    /**
     * 影子库标记
     */
    private static final String IS_SHADOW = "1";
    
    private final Map<String, String> parameters = new HashMap<>();
    private final Map<String, String> localValues = new HashMap<>();
 
    ...
    public static void clear() {
        CONTEXT.remove();
    }

    public static void setShadow() {
        CONTEXT.get().parameters.put(SHADOW, IS_SHADOW);
    }

    public static boolean isShadow() {
        return IS_SHADOW.equals(CONTEXT.get().parameters.get(SHADOW));
    }

}

yml文件多配置一倍数据库实例作为影子库

yaml 复制代码 
dataSources:
  ds_1:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
  ds_2:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
  ds_3:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  ds_4:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0

- !SHARDING
  tables:
    point_balance:
      actualDataNodes: ds_${1..4}.point_balance${1..2}
      databaseStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: d_uid_inline
      tableStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: t_uid_inline
      auditStrategy:
        auditorNames:
          - sharding_key_required_auditor
        allowHintDisable: true

CodeDbSharding和UIDDbSharding获取分库索引时加上偏移量

java 复制代码 
@Override
protected long getMode(long mode) {
    if (Context.isShadow()) {
        mode += 2;
    }
    return mode;
}

实现效果类似如下

java 复制代码 
@Test
public void testShadow() {
    Context.setShadow();
    List<CouponInfo> couponInfoList = couponInfoMapper.selectByCouponCode("1234567ABCD", "CN");
    Assert.assertTrue(couponInfoList.size() > 0);
    Context.clear();
    couponInfoList = couponInfoMapper.selectByCouponCode("1234567ABCD", "CN");
    Assert.assertTrue(couponInfoList.size() > 0);
}

执行结果如下,第一次带上标记落在ds_4,没带上影子库标记落在ds_2

yaml 复制代码 
2023-09-24 22:09:05.101|INFO |demo|1|127.0.0.1|1091ceaed1fe445a83a7265a6ef84065|Actual SQL: ds_4 ::: select id,         coupon_code,         rev_uid,         create_time,         modify_time,         country        from coupon_info1 where coupon_code=? and country=? ::: [1234567ABCD, CN]|ShardingSphere-SQL
<==    Columns: id, coupon_code, rev_uid, create_time, modify_time, country
<==        Row: 1, 1234567ABCD, 3, 2023-09-24 17:21:59.0, 2023-09-24 17:21:59.0, CN
<==        Row: 2, 1234567ABCD, 3, 2023-09-24 17:22:13.0, 2023-09-24 17:22:13.0, CN
<==        Row: 3, 1234567ABCD, 3, 2023-09-24 17:24:41.0, 2023-09-24 17:24:41.0, CN
<==        Row: 4, 1234567ABCD, 3, 2023-09-24 17:24:41.0, 2023-09-24 17:24:41.0, CN
<==      Total: 4
Closing non transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@79bf39e5]
Creating a new SqlSession
SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@354baab2] was not registered for synchronization because synchronization is not active
2023-09-24 22:09:05.184|DEBUG|demo|1|127.0.0.1|1091ceaed1fe445a83a7265a6ef84065|Fetching JDBC Connection from DataSource|org.springframework.jdbc.datasource.DataSourceUtils
JDBC Connection [org.apache.shardingsphere.driver.jdbc.core.connection.ShardingSphereConnection@30be6a05] will not be managed by Spring
==>  Preparing: select id, coupon_code, rev_uid, create_time, modify_time, country from coupon_info where coupon_code=? and country=? 
==> Parameters: 1234567ABCD(String), CN(String)
2023-09-24 22:09:05.184|INFO |demo|1|127.0.0.1|1091ceaed1fe445a83a7265a6ef84065|Logic SQL: select id,         coupon_code,         rev_uid,         create_time,         modify_time,         country        from coupon_info where coupon_code=? and country=?|ShardingSphere-SQL
2023-09-24 22:09:05.184|INFO |demo|1|127.0.0.1|1091ceaed1fe445a83a7265a6ef84065|Actual SQL: ds_2 ::: select id,         coupon_code,         rev_uid,         create_time,         modify_time,         country        from coupon_info1 where coupon_code=? and country=? ::: [1234567ABCD, CN]|ShardingSphere-SQL
<==    Columns: id, coupon_code, rev_uid, create_time, modify_time, country
<==        Row: 1, 1234567ABCD, 3, 2023-09-24 17:21:59.0, 2023-09-24 17:21:59.0, CN
<==        Row: 2, 1234567ABCD, 3, 2023-09-24 17:22:13.0, 2023-09-24 17:22:13.0, CN
<==        Row: 3, 1234567ABCD, 3, 2023-09-24 17:24:41.0, 2023-09-24 17:24:41.0, CN
<==        Row: 4, 1234567ABCD, 3, 2023-09-24 17:24:41.0, 2023-09-24 17:24:41.0, CN
<==      Total: 4
Closing non transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@354baab2]

针对海外某些国家，数据量较少，业务表也不进行分库分表处理

这块当前没有实现，需要在分库的路由算法增加国家维度，不同国家落到不同的库。这样使用yml配置不太便捷，需要用java代码来创建datasource

最终yml配置文件和整体代码结构

sharding-databases-tables.yaml文件如下

yaml 复制代码 
mode:
  type: Standalone
  repository:
    type: JDBC
dataSources:
  ds_1:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
  ds_2:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
  r_ds_1:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  r_ds_2:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  ds_3:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  ds_4:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  r_ds_3:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
  r_ds_4:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: -
    initialSize: 0
    minIdle: 0
rules:
- !READWRITE_SPLITTING
  dataSources:
    readwrite_ds1:
      staticStrategy:
        writeDataSourceName: ds_1
        readDataSourceNames:
          - r_ds_1
      loadBalancerName: coupon_db_random
    readwrite_ds2:
      staticStrategy:
        writeDataSourceName: ds_2
        readDataSourceNames:
          - r_ds_2
      loadBalancerName: coupon_db_random
  loadBalancers:
    coupon_db_random:
      type: RANDOM

- !SHARDING
  tables:
    point_balance:
      actualDataNodes: ds_${1..4}.point_balance${1..2}
      databaseStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: d_uid_inline
      tableStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: t_uid_inline
      auditStrategy:
        auditorNames:
          - sharding_key_required_auditor
        allowHintDisable: true
    point_balance_his:
      actualDataNodes: ds_${1..4}.point_balance_his${1..2}
      databaseStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: d_uid_inline
      tableStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: t_uid_inline
      auditStrategy:
        auditorNames:
          - sharding_key_required_auditor
        allowHintDisable: true
    param_config:
      actualDataNodes: ds_1.param_config
      databaseStrategy:
        none:
      tableStrategy:
        none:
    coupon_info:
      actualDataNodes: ds_${1..4}.coupon_info${1..2}
      databaseStrategy:
        standard:
          shardingColumn: coupon_code
          shardingAlgorithmName: d_code_inline
      tableStrategy:
        standard:
          shardingColumn: coupon_code
          shardingAlgorithmName: t_code_inline
      auditStrategy:
        auditorNames:
          - sharding_key_required_auditor
        allowHintDisable: true
  defaultShardingColumn: uid
  bindingTables:
    - point_balance,point_balance_his
    - coupon_info
  defaultDatabaseStrategy:
    none:
  defaultTableStrategy:
    none:
  shardingAlgorithms:
    d_uid_inline:
      type: CLASS_BASED
      props:
        algorithmClassName: com.toby.sharding.jdbc.source.start.db.sharding.UIDDbSharding
        strategy: STANDARD
    t_uid_inline:
      type: CLASS_BASED
      props:
        algorithmClassName: com.toby.sharding.jdbc.source.start.db.sharding.UIDTableSharding
        strategy: STANDARD
    d_code_inline:
      type: CLASS_BASED
      props:
        algorithmClassName: com.toby.sharding.jdbc.source.start.db.sharding.CodeDbSharding
        strategy: STANDARD
    t_code_inline:
      type: CLASS_BASED
      props:
        algorithmClassName: com.toby.sharding.jdbc.source.start.db.sharding.CodeTableSharding
        strategy: STANDARD
  auditors:
    sharding_key_required_auditor:
      type: DML_SHARDING_CONDITIONS
#  - !BROADCAST
#    tables:
#      -
props:
  sql-show: true

项目文件结构如下

项目代码在git目录，github.com/hongyuwen/H...

相关推荐
SomeB1oody3 分钟前
【Rust自学】7.3. use关键字 Pt.1：use的使用与as关键字
开发语言·后端·rust
minstbe13 分钟前
WEB开发 - Flask 入门：Jinja2 模板语法进阶 Python
后端·python·flask
无名之逆39 分钟前
lombok-macros
开发语言·windows·后端·算法·面试·rust·大学期末
m0_748247801 小时前
SpringBoot集成Flowable
java·spring boot·后端
散一世繁华，颠半世琉璃1 小时前
SpringBoot揭秘：URL与HTTP方法如何定位到Controller
spring boot·后端·http
web135085886352 小时前
9. 大数据集群(PySpark)+Hive+MySQL+PyEcharts+Flask：信用贷款风险分析与预测
大数据·hive·mysql
Y.O.U..2 小时前
Mysq学习-Mysql查询（4）
数据库·学习·mysql
安晴晚风2 小时前
从0开始在linux服务器上部署SpringBoot和Vue
linux·运维·前端·数据库·后端·运维开发
qq_321665335 小时前
mysql 数据库迁移到达梦数据库
数据库·mysql
海绵波波1078 小时前
flask后端开发（10）：问答平台项目结构搭建
后端·python·flask
热门推荐
01新手学习VR全景需要知道的几个问题02Ubuntu 20.04使用Livox mid 360 测试 FAST_LIO03玄机平台应急响应—webshell查杀04RAG 实践- Ollama+RagFlow 部署本地知识库05优化手机性能，解决卡顿问题：关闭这3个微信开关，释放内存空间06HCIA-datacom数通题库和录播视频资料07docker安装启动问题解决排查08【一文读懂】NTN（非地面网络）技术介绍09支付宝短视频平台创作分成激励项目10组基轨迹建模 GBTM的介绍与实现（Stata 或 R）