MYSQL系列-分库分表(三):Sharding-JDBC实现分库分表落地实践-中

系列文档参考 MYSQL系列-整体架构介绍

紧接上文MYSQL系列-分库分表(三):Sharding-JDBC实现分库分表落地实践-上

详细设计

表模型改动

当前表结构如下

point_shard1 point_shard2 分库分表字段
point_balance point_balance1 point_balance2 point_balance1 point_balance2 uid(long)
point_balance point_balance1 point_balance2 point_balance1 point_balance2 uid(long)
point_balance_his point_balance_his1 point_balance_his2 point_balance_his1 point_balance_his2 uid(long)
coupon_info coupon_info1 coupon_info2 coupon_info1 coupon_info2 coupon_code(String)

上述改动主要是将不同类型分库分表、以及配置表落在单库都涉及到

相关表结构如下

sql 复制代码
DROP TABLES IF EXISTS point_balance_his1;
CREATE TABLE IF NOT EXISTS `point_balance_his1` (
    `id` BIGINT(20) NOT NULL AUTO_INCREMENT COMMENT '主键ID',
    `uid` BIGINT(20) NOT NULL COMMENT 'uid',
    `point_sum` BIGINT(20) NOT NULL COMMENT '积分总和',
    `create_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
    `country` VARCHAR(20) NOT NULL DEFAULT 'CN' COMMENT '国家',
    PRIMARY KEY (`id`),
    INDEX idx_uid(`uid`)
    ) ENGINE=INNODB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COMMENT='用户积分变动历史表1';

DROP TABLES IF EXISTS point_balance_his2;
CREATE TABLE IF NOT EXISTS `point_balance_his2` (
    `id` BIGINT(20) NOT NULL AUTO_INCREMENT COMMENT '主键ID',
    `uid` BIGINT(20) NOT NULL COMMENT 'uid',
    `point_sum` BIGINT(20) NOT NULL COMMENT '积分总和',
    `create_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
    `country` VARCHAR(20) NOT NULL DEFAULT 'CN' COMMENT '国家',
    PRIMARY KEY (`id`),
    INDEX idx_uid(`uid`)
    ) ENGINE=INNODB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COMMENT='用户积分变动历史表2';

DROP TABLE IF EXISTS `coupon_info1`;
CREATE TABLE coupon_info1 (
                              `id` INT AUTO_INCREMENT NOT NULL COMMENT '自增ID',
                              `coupon_code` VARCHAR (20) NOT NULL COMMENT '券码CODE',
                              `rev_uid` BIGINT(20) DEFAULT NULL COMMENT '领券UID',
                              `create_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
                              `modify_time` TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
                              `country` VARCHAR (20) NOT NULL DEFAULT 'CN' COMMENT '国家',
                              PRIMARY KEY (`id`),
                              INDEX idx_coupon_code_country (coupon_code, country)
) ENGINE = INNODB AUTO_INCREMENT = 1 DEFAULT CHARSET = utf8 COMMENT = '券码表1' ;
DROP TABLE IF EXISTS `coupon_info2`;
CREATE TABLE coupon_info2 (
                              `id` INT AUTO_INCREMENT NOT NULL COMMENT '自增ID',
                              `coupon_code` VARCHAR (20) NOT NULL COMMENT '券码CODE',
                              `rev_uid` BIGINT(20) DEFAULT NULL COMMENT '领券UID',
                              `create_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
                              `modify_time` TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
                              `country` VARCHAR (20) NOT NULL DEFAULT 'CN' COMMENT '国家',
                              PRIMARY KEY (`id`),
                              INDEX idx_coupon_code_country (coupon_code, country)
) ENGINE = INNODB AUTO_INCREMENT = 1 DEFAULT CHARSET = utf8 COMMENT = '券码表2' ;

maven引用

主要涉及将sharding-jdbc等引用进来

注意,本期是基于sharding-jdbc 5.2来进行落地实践的

xml 复制代码
<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
        <version>2.7.9</version>
        <exclusions>
            <exclusion>
                <groupId>org.yaml</groupId>
                <artifactId>snakeyaml</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-jdbc</artifactId>
        <version>2.7.9</version>
    </dependency>
    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
        <version>5.1.49</version>
    </dependency>
    <dependency>
        <groupId>org.mybatis.spring.boot</groupId>
        <artifactId>mybatis-spring-boot-starter</artifactId>
        <version>2.1.0</version>
    </dependency>
    <dependency>
        <groupId>com.alibaba</groupId>
        <artifactId>druid</artifactId>
        <version>1.2.16</version>
    </dependency>
    <dependency>
        <groupId>org.aspectj</groupId>
        <artifactId>aspectjrt</artifactId>
        <version>1.9.5</version>
    </dependency>
    <dependency>
        <groupId>org.aspectj</groupId>
        <artifactId>aspectjweaver</artifactId>
        <version>1.9.5</version>
    </dependency>
    <dependency>
        <groupId>cglib</groupId>
        <artifactId>cglib-nodep</artifactId>
        <version>3.2.10</version>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.13</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <version>2.7.9</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>com.hyw.study</groupId>
        <artifactId>com.toby.logback</artifactId>
        <version>1.0-SNAPSHOT</version>
    </dependency>
    <dependency>
        <groupId>org.apache.shardingsphere</groupId>
        <artifactId>shardingsphere-jdbc-core-spring-boot-starter</artifactId>
        <version>5.2.0</version>
    </dependency>
    <dependency>
        <groupId>org.yaml</groupId>
        <artifactId>snakeyaml</artifactId>
        <version>1.33</version>
    </dependency>
    <dependency>
        <groupId>org.glassfish.jaxb</groupId>
        <artifactId>jaxb-runtime</artifactId>
        <version>2.3.8</version>
    </dependency>
</dependencies>

关键技术点实现

整体框架

META-INF新建sharding-databases-tables.yaml配置文件,主要将druid数据源配置、分库分表算法、读写分离等放进去 然后使用JAVA代码加载创建datasource

java 复制代码
@Configuration
public class ShardingSphereConfig {

    @Bean
    public DataSource dataSource() throws IOException, SQLException {
        DataSource dataSource = YamlShardingSphereDataSourceFactory.createDataSource(
                getFile("classpath:META-INF/sharding-databases-tables.yaml"));
        return dataSource;
    }
}

mybatics相关配置如下 application.yml配置

yaml 复制代码
spring:
  main:
    allow-bean-definition-overriding: true
mybatis:
  config-location: classpath:sqlmap-config.xml

sqlmap-config.xml如下

java 复制代码
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE configuration
        PUBLIC "-//mybatis.org//DTD Config 3.0//EN"
        "http://mybatis.org/dtd/mybatis-3-config.dtd">

<configuration>
    <settings>
        <!-- 开启驼峰命名自动映射 -->
        <setting name="mapUnderscoreToCamelCase" value="true"/>
        <!-- 开启二级缓存 -->
        <setting name="cacheEnabled" value="true"/>
        <!-- 打印 SQL 语句 -->
        <setting name="logImpl" value="STDOUT_LOGGING"/>
    </settings>
</configuration>

Mapper文件本次采用注解的方式,去掉xml配置,具体如下:

java 复制代码
public interface CouponInfoMapper {

    @Insert("insert into coupon_info (coupon_code, rev_uid, country)\n" +
            "        values (#{couponCode}, #{recUid}, #{country})\n")
    int addCouponCode(CouponInfo couponInfo);

    @Select("select id,\n" +
            "        coupon_code,\n" +
            "        rev_uid,\n" +
            "        create_time,\n" +
            "        modify_time,\n" +
            "        country" +
            "        from coupon_info where coupon_code=#{couponCode} and country=#{country}")
    List<CouponInfo> selectByCouponCode(@Param("couponCode") String couponCode, @Param("country") String country);
}

业务表point_balance1基于UID进行分库分表

yml文件中配置数据源

yml 复制代码
dataSources:
  ds_1:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard1?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: 1Qaz1Qaz
  ds_2:
    dataSourceClassName: com.alibaba.druid.pool.DruidDataSource
    driverClassName: com.mysql.jdbc.Driver
    url: jdbc:mysql://172.17.108.172:3306/point_shard2?autoReconnect=true&characterEncoding=UTF-8&useUnicode=true&connectTimeout=3000&socketTimeout=3000
    username: root
    password: 1Qaz1Qaz

主要druid是url,连接池是HikariCP的化是jdbcUrl

配置分库分表,具体值含义参考官网-数据源配置

yml 复制代码
- !SHARDING
  tables:
    point_balance:
      actualDataNodes: ds_${1..4}.point_balance${1..2}
      databaseStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: d_uid_inline
      tableStrategy:
        standard:
          shardingColumn: uid
          shardingAlgorithmName: t_uid_inline
      auditStrategy:
        auditorNames:
          - sharding_key_required_auditor
        allowHintDisable: true

d_uid_inline是分库算法

yml 复制代码
shardingAlgorithms:
  d_uid_inline:
    type: CLASS_BASED
    props:
      algorithmClassName: com.toby.sharding.jdbc.source.start.db.sharding.UIDDbSharding
      strategy: STANDARD

具体的分库逻辑在com.toby.sharding.jdbc.source.start.db.sharding.UIDDbSharding

java 复制代码
public abstract class AbstractLongShardingAlgorithm implements StandardShardingAlgorithm<Long> {
    Properties properties;

    @Override
    public String doSharding(Collection<String> collection, PreciseShardingValue<Long> preciseShardingValue) {
        String name = routeTable(collection, getLongFromObject(preciseShardingValue.getValue()));
        if (Objects.isNull(name)) {
            throw new UnsupportedOperationException("cannot not find rout for" + preciseShardingValue);
        }
        //返回需要分库的yml中配置名字
        return name;
    }

    @Override
    public Collection<String> doSharding(Collection<String> collection, RangeShardingValue<Long> rangeShardingValue) {
        Range<Long> range = rangeShardingValue.getValueRange();
        Set<String> names = new HashSet<>();
        for (long start = range.lowerEndpoint(); start <= range.upperEndpoint(); start++) {
            String name = routeTable(collection, start);
            if (Objects.isNull(name)) {
                throw new UnsupportedOperationException("cannot not find range rout for " + start);
            }
            names.add(name);
        }
        return names;
    }

    @Override
    public Properties getProps() {
        return properties;
    }

    @Override
    public void init(Properties properties) {
        this.properties = properties;
    }

    private long getLongFromObject(Object obj) {
        long value = -1L;
        try {
            if (obj instanceof Integer) {
                value = ((Integer) obj).longValue();
            } else if (obj instanceof BigInteger) {
                value = ((BigInteger) obj).longValue();
            } else if (obj instanceof String) {
                value = Long.parseLong((String) obj);
            } else {
                value = Long.parseLong(obj.toString());
            }
        } catch (Exception e) {
            //ingore
        }
        return value;
    }

    private String routeTable(Collection<String> collection, long value) {
        long mod = getValue(value);
        mod = getMode(mod);
        for (String name : collection) {
            if (name.endsWith(String.valueOf(mod))) {
                return name;
            }
        }
        return null;
    }

    protected abstract long getValue(long value);

    protected long getMode(long mode) {
        return mode;
    }
}
java 复制代码
public class UIDDbSharding extends AbstractLongShardingAlgorithm {
    @Override
    protected long getValue(long value) {
        //具体分库算法
        return value / 2 % 2 + 1;
    }

    @Override
    protected long getMode(long mode) {
        //影子库路由算法,后面讲
        if (Context.isShadow()) {
            mode += 2;
        }
        return mode;
    }
}

t_uid_inline是分表算法

yml 复制代码
t_uid_inline:
  type: CLASS_BASED
  props:
    algorithmClassName: com.toby.sharding.jdbc.source.start.db.sharding.UIDTableSharding
    strategy: STANDARD

com.toby.sharding.jdbc.source.start.db.sharding.UIDTableSharding存放具体分表逻辑

java 复制代码
public class UIDTableSharding extends AbstractLongShardingAlgorithm {

    @Override
    protected long getValue(long value) {
        return value % 2 + 1;
    }
}

综上是point_balance基于shardingjdbc的分库分表配置项

配置表param_config默认走point_shard1库,不进行分库分表

可以在yml文件配置如下,代表既不分库也不分表

java 复制代码
param_config:
  actualDataNodes: ds_1.param_config
  databaseStrategy:
    none:
  tableStrategy:
    none:

当然也可以什么都不配置,sharding可以自己找到对应的数据库实例,这块没详细看,亲测如果我把配置表放在point_shard2,如果param_config啥都不配置,也是可以获取到的

未完待续

相关推荐
程序员爱钓鱼5 分钟前
Go语言实战案例-项目实战篇:新闻聚合工具
后端·google·go
IT_陈寒6 分钟前
Python开发者必须掌握的12个高效数据处理技巧,用过都说香!
前端·人工智能·后端
一只叫煤球的猫8 小时前
写代码很6,面试秒变菜鸟?不卖课,面试官视角走心探讨
前端·后端·面试
bobz9659 小时前
tcp/ip 中的多路复用
后端
bobz9659 小时前
tls ingress 简单记录
后端
你的人类朋友10 小时前
什么是OpenSSL
后端·安全·程序员
bobz96510 小时前
mcp 直接操作浏览器
后端
程序新视界11 小时前
MySQL中什么是回表查询,如何避免和优化?
mysql
前端小张同学13 小时前
服务器部署 gitlab 占用空间太大怎么办,优化思路。
后端
databook13 小时前
Manim实现闪光轨迹特效
后端·python·动效