系列文档参考 MYSQL系列-整体架构介绍

本文紧接上文 MYSQL系列-分库分表（二）：Spring动态数据源实现分库分表落地实践-上

实现方案

分库核心实现

从上文的UML类图可以看出，分库核心实现包括两部分：

在执行时将数据源保存在threadLocal的变量中
在Spring动态数据源寻找本次数据源时，将其从threadLocal中取出来，通过determineCurrentLookupKey返回

Mapper文件执行前将数据源放到threadLocal的变量中

具体代码在ShardingDataSourceAspect中，首先设置切点

java 复制代码

@Pointcut("execution(* com.toby.dynamic.data.source.db.dao.*.*(..))")
public void daoAspect() {

}

然后在com.toby.dynamic.data.source.db.dao.*.*(..))执行前先切换数据源

java 复制代码

@Before("daoAspect()")
public void changeDataSource(JoinPoint point) {
    String dataSource;
    Signature signature = point.getSignature();
    Annotation[][] annotations = ((MethodSignature) signature).getMethod().getParameterAnnotations();
    Object[] params = point.getArgs();
    if (annotations.length > 0 && annotations[0].length > 0) {
        Map<String, Object> paramMap = getParamMap(annotations, params);
        dataSource = dbRouter.getDataSource(paramMap);
    } else {
        dataSource = dbRouter.getDataSource(params[0]);
    }
    LOGGER.info("changeDataSource to data source={}", dataSource);
    SharingDataSourceHolder.setRouterDataSource(dataSource);
}

private Map<String, Object> getParamMap(Annotation[][] annotations, Object[] params) {
    Map<String, Object> paramMap = new HashMap<>();
    for (int i = 0; i < annotations.length; i++) {
        Annotation[] annos = annotations[i];
        if (Objects.isNull(annos) || annos.length <= 0) {
            continue;
        }
        Annotation annotation = annotations[i][0];
        //Params是mybatics的注解
        if (annotation instanceof Param) {
            paramMap.put(((Param) annotation).value(), params[i]);
        }
    }
    return paramMap;
}

dbRouter.getDataSource会调到ContextDbRouter中

java 复制代码

@Override
public String getDataSource(Object param) {
    Map<Integer, String> dataSourceList = getDataSourceList();
    int index = DataSourceUtil.shardDB(param);
    if (index < 0) {
        return getDefaultDataSource();
    }
    return dataSourceList.get(index);
}

getDataSourceList获取到读库或者写库数据源集合

java 复制代码

private Map<Integer, String> getDataSourceList() {
    return dataSourceGroup.get(getGroupName());
}

private String getGroupName() {
    //扩展，后续支持读写库
    if (Context.get(DataSourceUtil.READ, "0").equals("1")) {
        return READ;
    }
    return NORMAL;
}

getGroupName中包含读写库的操作，后面会将；DataSourceUtil.shardDB会获取分库路由算法

java 复制代码

public static int shardDB(Object param) {
    long uid = DataSourceUtil.getUid(param);
    if (uid < 0) {
        return -1;
    }
    return Math.toIntExact(uid % SHARD_DB) + 1;
}

getUid主要是通过入参中的各种类型，基本类型、Map、List、对象中成员变量获取uid值

java 复制代码

private static long getUid(Object param) {
    if (isBasicType(param)) {
        return Long.parseLong(param.toString());
    }
    if (param instanceof List) {
        return parseObject(((List) param).get(0));
    }
    if (param instanceof Map) {
        return Long.parseLong(((Map) param).getOrDefault(UID, NOT_FOUND).toString());
    }
    return parseObject(param);

}

private static boolean isBasicType(Object param) {
    Class paramCls = param.getClass();
    return paramCls == String.class || paramCls == long.class
            || paramCls == int.class || paramCls == Long.class || paramCls == Integer.class
            || paramCls == BigInteger.class;
}

private static long parseObject(Object param) {
    PropertyDescriptor propertyDescriptor = BeanUtils.getPropertyDescriptor(param.getClass(), UID);
    if (Objects.isNull(propertyDescriptor) || Objects.isNull(propertyDescriptor.getReadMethod())) {
        return NOT_FOUND;
    }
    try {
        Object result = propertyDescriptor.getReadMethod().invoke(param);
        if (Objects.isNull(result)) {
            return NOT_FOUND;
        }
        return Long.parseLong(result.toString());
    } catch (InvocationTargetException | IllegalAccessException e) {
        LOGGER.error("parseObject error", e);
        return NOT_FOUND;
    }
}

SharingDataSourceHolder.setRouterDataSource是将数据源放到threadlocal的线程变量中

java 复制代码

private static ThreadLocal<String> ROUTER_DATASOURCE = new ThreadLocal<>();

public static void setRouterDataSource(String dataSource) {
    ROUTER_DATASOURCE.set(dataSource);
}

Spring动态数据源切换处理

主要逻辑在determineCurrentLookupKey

java 复制代码

protected Object determineCurrentLookupKey() {
    String dataSource = SharingDataSourceHolder.getRouterDataSource();
    LOGGER.info("dataSource={}", dataSource);
    if (Objects.isNull(dataSource)) {
        return dbRouter.getDefaultDataSource();
    }
    return dataSource;
}

在启动初始化的时候会将数据源列表放到AbstractRoutingDataSource的targetDataSources

java 复制代码

public void afterPropertiesSet() {
    Map<String, Map<Integer,String>> dataSourceGroup = dbRouter.getDataSourceGroup();
    Map<Object, Object> targetDataSourceMap = new HashMap<>();
    for (Map.Entry<String, Map<Integer,String>> entry : dataSourceGroup.entrySet()) {
        Map<Integer,String> dataSourceList = entry.getValue();
        for (Map.Entry<Integer,String> dataSourceName : dataSourceList.entrySet()) {
            Object dataSource = applicationContext.getBean(dataSourceName.getValue());
            if (Objects.isNull(dataSource) || !(dataSource instanceof DataSource)) {
                throw new IllegalStateException("cannot find data soure name:" + dataSourceName);
            }
            targetDataSourceMap.put(dataSourceName.getValue(), dataSource);
        }
    }
    super.setTargetDataSources(targetDataSourceMap);
    super.afterPropertiesSet();
}

综上，将分库原理实现讲解完毕

分表实现方案

分表实现方案采用mybatics的插件来进行处理，整体思路是新增一个分表注解TableSplit，里面包含分要的分表

java 复制代码

@Target({ElementType.METHOD, ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
public @interface TableSplit {

    boolean split() default true;

    String value();
}

新增一个针对prepare处理的切面类，将执行SQL里面表point_balance替换为具体的表point_balance1或者point_balance2

主要逻辑在TableSplitInterceptor

java 复制代码

@Intercepts(@Signature(method = "prepare", type = StatementHandler.class, args = {Connection.class, Integer.class}))
public class TableSplitInterceptor implements Interceptor {
    private static final Logger LOGGER = LoggerFactory.getLogger(TableSplitInterceptor.class);

    @Override
    public Object intercept(Invocation invocation) throws Throwable {
        StatementHandler statementHandler = (StatementHandler) invocation.getTarget();
        MetaObject metaObject = SystemMetaObject.forObject(statementHandler);
        doSplitTable(metaObject);
        return invocation.proceed();
    }
    ...

doSplitTable是分表主要逻辑处理类，主要是通过入参找到分表的index，然后替换到delegate.boundSql.sql

java 复制代码

private void doSplitTable(MetaObject metaObject) throws ClassNotFoundException {
    Object parameterObject = metaObject.getValue("delegate.boundSql.parameterObject");
    String originalSql = (String) metaObject.getValue("delegate.boundSql.sql");
    if (Objects.isNull(originalSql) || originalSql.length() <= 0) {
        LOGGER.info("originalSql is null,return.");
        return;
    }
    MappedStatement mappedStatement = (MappedStatement) metaObject.getValue("delegate.mappedStatement");
    String id = mappedStatement.getId();
    String cls = id.substring(0, id.lastIndexOf("."));
    Class<?> clsObj = Class.forName(cls);
    TableSplit tableSplit = clsObj.getAnnotation(TableSplit.class);
    if (Objects.isNull(tableSplit) || !tableSplit.split()) {
        return;
    }
    int tableNums = DataSourceUtil.shardTable(parameterObject);
    String[] tableNames = tableSplit.value().split(";");
    String relTableName = null;
    for (String name : tableNames) {
        relTableName = name + tableNums;
        originalSql = originalSql.replaceAll(name, relTableName);
    }
    metaObject.setValue("delegate.boundSql.sql", originalSql);
}

分表的逻辑如下shardTable

java 复制代码

public static int shardTable(Object param) {
    long uid = getUid(param);
    if (uid < 0) {
        return 0;
    }
    long mod = uid / SHARD_DB;
    return Math.toIntExact(mod % SHARD_TABLE) + 1;
}

使用时，需要在对应的dao类加上注解即可，如下所示

java 复制代码

@TableSplit(value = "point_balance")
public interface UserPointBalanceMapper {

    int addUserPointBalance(UserPointBalance userPointBalance);
    ...

读写分离实现

读写分离主要点是在执行dao时，打上一个标签，告诉程序当前是从读库查询还是从写库执行即可在配置文件中，我们已经将读写组分开，放到ContextDbRouter.dataSourceGroup

java 复制代码

<util:map id="datasourceMap" key-type="java.lang.String" value-type="java.util.Map">
    <entry key="MASTER">
        <map key-type="java.lang.String" value-type="java.lang.String">
            <entry key="1" value="db_w_1"/>
            <entry key="2" value="db_w_2"/>
            <!-- default-->
            <entry key="-1" value="db_w_1"/>
        </map>
    </entry>
    <entry key="READ">
        <map>
            <entry key="1" value="db_r_1"/>
            <entry key="2" value="db_r_2"/>
            <!-- default-->
            <entry key="-1" value="db_r_1"/>
        </map>
    </entry>
</util:map>

新增一个读的注解ReadDataSource

java 复制代码

@Target({ElementType.METHOD, ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
public @interface ReadDataSource {
}

同时新增一个切面ReadDataSourceAspect

java 复制代码

@Before("@annotation(readDataSource)")
public void addReadDataSource(JoinPoint joinPoint, ReadDataSource readDataSource) {
    LOGGER.info("addReadDataSource begin.");
    Context.set(DataSourceUtil.READ, "1");
}

@After("@annotation(readDataSource)")
public void removeReadDataSource(JoinPoint joinPoint, ReadDataSource readDataSource) {
    LOGGER.info("addReadDataSource end.");
    Context.remove(DataSourceUtil.READ);
}

最终在获取数据源分组时判读是返回读的分组还是写分组数据源，代码在ContextDbRouter

java 复制代码

private String getGroupName() {
    //扩展，后续支持读写库
    if (Context.get(DataSourceUtil.READ, "0").equals("1")) {
        return READ;
    }
    return NORMAL;
}

使用时加上对应的注解即可，如UserPointBalanceMapper

java 复制代码

@ReadDataSource
List<UserPointBalance> selectUserPointBalanceByUidAndCountry(@Param("uid") BigInteger uid, @Param("country") String country);

配置表走默认库

如果配置表不采用广播表的方式，需要放到指定库

实际业务场景中，有些业务量不是很大的数据，比如一些管理台审计日志，本身不属于配置表的范畴，也不需要分库分表，也是需要单独某个库存储的

首先在mybatics配置中增加默认库的配置

java 复制代码

<util:map id="datasourceMap" key-type="java.lang.String" value-type="java.util.Map">
    <entry key="MASTER">
        <map key-type="java.lang.String" value-type="java.lang.String">
            <entry key="1" value="db_w_1"/>
            <entry key="2" value="db_w_2"/>
            <!-- default-->
            <entry key="-1" value="db_w_1"/>
        </map>
    </entry>
    <entry key="READ">
        <map>
            <entry key="1" value="db_r_1"/>
            <entry key="2" value="db_r_2"/>
            <!-- default-->
            <entry key="-1" value="db_r_1"/>
        </map>
    </entry>
</util:map>

将配置表的mapper文件和到文件单独文件夹存放，保证ShardingDataSourceAspect切面逻辑不执行