MYSQL系列-分库分表（三）：Sharding-JDBC实现分库分表落地实践-中

技术驿站2023-09-24 21:55

系列文档参考 MYSQL系列-整体架构介绍

紧接上文MYSQL系列-分库分表（三）：Sharding-JDBC实现分库分表落地实践-上

详细设计

表模型改动

当前表结构如下

point_shard1 point_shard2 分库分表字段
point_balance point_balance1 point_balance2 point_balance1 point_balance2 uid(long)
point_balance point_balance1 point_balance2 point_balance1 point_balance2 uid(long)
point_balance_his point_balance_his1 point_balance_his2 point_balance_his1 point_balance_his2 uid(long)
coupon_info coupon_info1 coupon_info2 coupon_info1 coupon_info2 coupon_code(String)

上述改动主要是将不同类型分库分表、以及配置表落在单库都涉及到

相关表结构如下

sql 复制代码 
DROP TABLES IF EXISTS point_balance_his1;
CREATE TABLE IF NOT EXISTS `point_balance_his1` (
    `id` BIGINT(20) NOT NULL AUTO_INCREMENT COMMENT '主键ID',
    `uid` BIGINT(20) NOT NULL COMMENT 'uid',
    `point_sum` BIGINT(20) NOT NULL COMMENT '积分总和',
    `create_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
    `country` VARCHAR(20) NOT NULL DEFAULT 'CN' COMMENT '国家',
    PRIMARY KEY (`id`),
    INDEX idx_uid(`uid`)
    ) ENGINE=INNODB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COMMENT='用户积分变动历史表1';

DROP TABLES IF EXISTS point_balance_his2;
CREATE TABLE IF NOT EXISTS `point_balance_his2` (
    `id` BIGINT(20) NOT NULL AUTO_INCREMENT COMMENT '主键ID',
    `uid` BIGINT(20) NOT NULL COMMENT 'uid',
    `point_sum` BIGINT(20) NOT NULL COMMENT '积分总和',
    `create_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
    `country` VARCHAR(20) NOT NULL DEFAULT 'CN' COMMENT '国家',
    PRIMARY KEY (`id`),
    INDEX idx_uid(`uid`)
    ) ENGINE=INNODB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COMMENT='用户积分变动历史表2';

DROP TABLE IF EXISTS `coupon_info1`;
CREATE TABLE coupon_info1 (
                              `id` INT AUTO_INCREMENT NOT NULL COMMENT '自增ID',
                              `coupon_code` VARCHAR (20) NOT NULL COMMENT '券码CODE',
                              `rev_uid` BIGINT(20) DEFAULT NULL COMMENT '领券UID',
                              `create_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
                              `modify_time` TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
                              `country` VARCHAR (20) NOT NULL DEFAULT 'CN' COMMENT '国家',
                              PRIMARY KEY (`id`),
                              INDEX idx_coupon_code_country (coupon_code, country)
) ENGINE = INNODB AUTO_INCREMENT = 1 DEFAULT CHARSET = utf8 COMMENT = '券码表1' ;
DROP TABLE IF EXISTS `coupon_info2`;
CREATE TABLE coupon_info2 (
                              `id` INT AUTO_INCREMENT NOT NULL COMMENT '自增ID',
                              `coupon_code` VARCHAR (20) NOT NULL COMMENT '券码CODE',
                              `rev_uid` BIGINT(20) DEFAULT NULL COMMENT '领券UID',
                              `create_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
                              `modify_time` TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
                              `country` VARCHAR (20) NOT NULL DEFAULT 'CN' COMMENT '国家',
                              PRIMARY KEY (`id`),
                              INDEX idx_coupon_code_country (coupon_code, country)
) ENGINE = INNODB AUTO_INCREMENT = 1 DEFAULT CHARSET = utf8 COMMENT = '券码表2' ;

maven引用

主要涉及将sharding-jdbc等引用进来

注意，本期是基于sharding-jdbc 5.2来进行落地实践的

xml 复制代码 
<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
        <version>2.7.9</version>
        <exclusions>
            <exclusion>
                <groupId>org.yaml</groupId>
                <artifactId>snakeyaml</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-jdbc</artifactId>
        <version>2.7.9</version>
    </dependency>
    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
        <version>5.1.49</version>
    </dependency>
    <dependency>
        <groupId>org.mybatis.spring.boot</groupId>
        <artifactId>mybatis-spring-boot-starter</artifactId>
        <version>2.1.0</version>
    </dependency>
    <dependency>
        <groupId>com.alibaba</groupId>
        <artifactId>druid</artifactId>
        <version>1.2.16</version>
    </dependency>
    <dependency>
        <groupId>org.aspectj</groupId>
        <artifactId>aspectjrt</artifactId>
        <version>1.9.5</version>
    </dependency>
    <dependency>
        <groupId>org.aspectj</groupId>
        <artifactId>aspectjweaver</artifactId>
        <version>1.9.5</version>
    </dependency>
    <dependency>
        <groupId>cglib</groupId>
        <artifactId>cglib-nodep</artifactId>
        <version>3.2.10</version>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.13</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <version>2.7.9</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>com.hyw.study</groupId>
        <artifactId>com.toby.logback</artifactId>
        <version>1.0-SNAPSHOT</version>
    </dependency>
    <dependency>
        <groupId>org.apache.shardingsphere</groupId>
        <artifactId>shardingsphere-jdbc-core-spring-boot-starter</artifactId>
        <version>5.2.0</version>
    </dependency>
    <dependency>
        <groupId>org.yaml</groupId>
        <artifactId>snakeyaml</artifactId>
        <version>1.33</version>
    </dependency>
    <dependency>
        <groupId>org.glassfish.jaxb</groupId>
        <artifactId>jaxb-runtime</artifactId>
        <version>2.3.8</version>
    </dependency>
</dependencies>

关键技术点实现

整体框架

在META-INF新建sharding-databases-tables.yaml配置文件，主要将druid数据源配置、分库分表算法、读写分离等放进去 然后使用JAVA代码加载创建datasource

java 复制代码 
@Configuration
public class ShardingSphereConfig {

    @Bean
    public DataSource dataSource() throws IOException, SQLException {
        DataSource dataSource = YamlShardingSphereDataSourceFactory.createDataSource(
                getFile("classpath:META-INF/sharding-databases-tables.yaml"));
        return dataSource;
    }
}

mybatics相关配置如下 application.yml配置

yaml 复制代码 
spring:
  main:
    allow-bean-definition-overriding: true
mybatis:
  config-location: classpath:sqlmap-config.xml

sqlmap-config.xml如下

java 复制代码 
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE configuration
        PUBLIC "-//mybatis.org//DTD Config 3.0//EN"
        "http://mybatis.org/dtd/mybatis-3-config.dtd">

<configuration>
    <settings>
        <!-- 开启驼峰命名自动映射 -->
        <setting name="mapUnderscoreToCamelCase" value="true"/>
        <!-- 开启二级缓存 -->
        <setting name="cacheEnabled" value="true"/>
        <!-- 打印 SQL 语句 -->
        <setting name="logImpl" value="STDOUT_LOGGING"/>
    </settings>
</configuration>

Mapper文件本次采用注解的方式，去掉xml配置，具体如下：

java 复制代码 
public interface CouponInfoMapper {

    @Insert("insert into coupon_info (coupon_code, rev_uid, country)\n" +
            "        values (#{couponCode}, #{recUid}, #{country})\n")
    int addCouponCode(CouponInfo couponInfo);

    @Select("select id,\n" +
            "        coupon_code,\n" +
            "        rev_uid,\n" +
            "        create_time,\n" +
            "        modify_time,\n" +
            "        country" +
            "        from coupon_info where coupon_code=#{couponCode} and country=#{country}")
    List<CouponInfo> selectByCouponCode(@Param("couponCode") String couponCode, @Param("country") String country);
}

业务表point_balance1基于UID进行分库分表

yml文件中配置数据源

yml 复制代码 
dataSources:
  ds_1:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: 1Qaz1Qaz
  ds_2:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: 1Qaz1Qaz

主要druid是url，连接池是HikariCP的化是jdbcUrl

配置分库分表，具体值含义参考官网-数据源配置

yml 复制代码 
- !SHARDING
  tables:
    point_balance:
      actualDataNodes: ds_${1..4}.point_balance${1..2}
      databaseStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: d_uid_inline
      tableStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: t_uid_inline
      auditStrategy:
        auditorNames:
          - sharding_key_required_auditor
        allowHintDisable: true

d_uid_inline是分库算法

yml 复制代码 
shardingAlgorithms:
  d_uid_inline:
    type: CLASS_BASED
    props:
      algorithmClassName: com.toby.sharding.jdbc.source.start.db.sharding.UIDDbSharding
      strategy: STANDARD

具体的分库逻辑在com.toby.sharding.jdbc.source.start.db.sharding.UIDDbSharding

java 复制代码 
public abstract class AbstractLongShardingAlgorithm implements StandardShardingAlgorithm<Long> {
    Properties properties;

    @Override
    public String doSharding(Collection<String> collection, PreciseShardingValue<Long> preciseShardingValue) {
        String name = routeTable(collection, getLongFromObject(preciseShardingValue.getValue()));
        if (Objects.isNull(name)) {
            throw new UnsupportedOperationException("cannot not find rout for" + preciseShardingValue);
        }
        //返回需要分库的yml中配置名字
        return name;
    }

    @Override
    public Collection<String> doSharding(Collection<String> collection, RangeShardingValue<Long> rangeShardingValue) {
        Range<Long> range = rangeShardingValue.getValueRange();
        Set<String> names = new HashSet<>();
        for (long start = range.lowerEndpoint(); start <= range.upperEndpoint(); start++) {
            String name = routeTable(collection, start);
            if (Objects.isNull(name)) {
                throw new UnsupportedOperationException("cannot not find range rout for " + start);
            }
            names.add(name);
        }
        return names;
    }

    @Override
    public Properties getProps() {
        return properties;
    }

    @Override
    public void init(Properties properties) {
        this.properties = properties;
    }

    private long getLongFromObject(Object obj) {
        long value = -1L;
        try {
            if (obj instanceof Integer) {
                value = ((Integer) obj).longValue();
            } else if (obj instanceof BigInteger) {
                value = ((BigInteger) obj).longValue();
            } else if (obj instanceof String) {
                value = Long.parseLong((String) obj);
            } else {
                value = Long.parseLong(obj.toString());
            }
        } catch (Exception e) {
            //ingore
        }
        return value;
    }

    private String routeTable(Collection<String> collection, long value) {
        long mod = getValue(value);
        mod = getMode(mod);
        for (String name : collection) {
            if (name.endsWith(String.valueOf(mod))) {
                return name;
            }
        }
        return null;
    }

    protected abstract long getValue(long value);

    protected long getMode(long mode) {
        return mode;
    }
}
java 复制代码 
public class UIDDbSharding extends AbstractLongShardingAlgorithm {
    @Override
    protected long getValue(long value) {
        //具体分库算法
        return value / 2 % 2 + 1;
    }

    @Override
    protected long getMode(long mode) {
        //影子库路由算法，后面讲
        if (Context.isShadow()) {
            mode += 2;
        }
        return mode;
    }
}

t_uid_inline是分表算法

yml 复制代码 
t_uid_inline:
  type: CLASS_BASED
  props:
    algorithmClassName: com.toby.sharding.jdbc.source.start.db.sharding.UIDTableSharding
    strategy: STANDARD

com.toby.sharding.jdbc.source.start.db.sharding.UIDTableSharding存放具体分表逻辑

java 复制代码 
public class UIDTableSharding extends AbstractLongShardingAlgorithm {

    @Override
    protected long getValue(long value) {
        return value % 2 + 1;
    }
}

综上是point_balance基于shardingjdbc的分库分表配置项

配置表param_config默认走point_shard1库，不进行分库分表

可以在yml文件配置如下，代表既不分库也不分表

java 复制代码 
param_config:
  actualDataNodes: ds_1.param_config
  databaseStrategy:
    none:
  tableStrategy:
    none:

当然也可以什么都不配置，sharding可以自己找到对应的数据库实例，这块没详细看，亲测如果我把配置表放在point_shard2，如果param_config啥都不配置，也是可以获取到的

未完待续

相关推荐
沐怡旸35 分钟前
【底层机制】LeakCanary深度解析：从对象监控到内存泄漏分析的完整技术体系
android·面试
王元_SmallA37 分钟前
【玩转全栈】----Django基本配置和介绍
java·后端
Moonbit1 小时前
招募进行时 | MoonBit AI : 程序语言 & 大模型
前端·后端·面试
AlianNiew1 小时前
从源码到实战：用 Java 打造“限时+防重放”的文件安全预览链接
java·后端
spencer_tseng1 小时前
mysql uuid()
mysql·uuid
uhakadotcom1 小时前
基于 TOON + Next.js 来大幅节省 token 并运行大模型
前端·面试·github
小威要向诸佬学习呀2 小时前
2025年软件外包避坑指南与平台推荐：开发者实用经验分享
后端
冒泡的肥皂2 小时前
2PL+MVCC看一些场景
数据库·后端·mysql
bcbnb2 小时前
Charles 抓不到包怎么办？一线工程师的排查与真机抓包流程
后端
岁岁岁平安2 小时前
python mysql-connector、PyMySQL基础
python·mysql·pymysql
热门推荐
01GitHub 镜像站点02《大数据技术原理与应用》实验报告三 熟悉HBase常用操作03UV安装并设置国内源04综合整理：pdf预览显示：你尝试预览的文件可能对你的计算机有害。如果你信任此文件以及其来源，请打开此文件以看其内容，如何解决以正常预览文件05BongoCat - 跨平台键盘猫动画工具06Linux下V2Ray安装配置指南07npm使用国内淘宝镜像的方法08jdk21下载、安装（Windows、Linux、macOS）09NVIDIA显卡驱动、CUDA、cuDNN 和 TensorRT 版本匹配指南10GitLab 零基础入门指南：从安装到项目管理全流程