Debezium兼容人大金仓KingBase8及达梦8

Debezium

Debezium: 一款国外开源的数据库实时捕获项目，目前最新版本为2023-10-10的2.4，支持主流的数据库包括：

• MySQL
• MongoDB
• SQL Server
• Db2
• Oracle
• PostgreSQL
• Cassandra
• Vitess
• Spanner

有两种部署模式，包括独立部署（server+kafka）以及嵌入式部署（内嵌到项目中）。

如何兼容人大金仓

通过考古国外人大金仓官网后发现该国产数据库是基于PostgreSQL研发的，于是试试直接用PostgreSQL的连接器是否可以运行。由于kingbase沿用了pg的复制槽，所以我们主要关注解码插件好了。 三种插件：

• pgoutput
• pgproto
• testdecoding

插件名	是否自带	可行性	安装方式
pgoutput	否	可行	编译部署
pgproto	否	由于无法获得.proto协议文件，可行性不大	无需安装
testdecoding	是	可行	无需安装

这里采用testdecoding插件，只需增加testdecoding解析代码即可，其他配置基本参照pg。

public enum MessageType {
    RELATION,
    BEGIN,
    COMMIT,
    INSERT,
    UPDATE,
    DELETE;

    public static MessageType forType(String context) {
        List<String> strings;
        if (Strings.isNullOrEmpty(context) || isEmpty(strings = Splitter.on(" ").splitToList(context))) {
            throw new InvalidParameterException("Invalid message type");
        }
        String first = strings.get(0);
        if ("BEGIN".equals(first)) {
            return BEGIN;
        }
        else if ("COMMIT".equals(first)) {
            return COMMIT;
        }
        switch (strings.get(2)) {
            case "INSERT:":
                return INSERT;
            case "UPDATE:":
                return UPDATE;
            case "DELETE:":
                return DELETE;
            default:
                return RELATION;
        }
    }
}

public TestDecodingMessageDecoder(MessageDecoderContext decoderContext, PostgresConnection connection) {
    this.decoderContext = decoderContext;
    this.connection = connection;
}

如何兼容达梦

通过挖掘国内达梦官网后发现其基本基于oracle研发，许多概念都与oracle相似，惊觉走kingbase的套路或许可行。 <math xmlns="http://www.w3.org/1998/Math/MathML"> 不得不说国产数据库为了让我们快速上手也是绞尽脑汁！ 不得不说国产数据库为了让我们快速上手也是绞尽脑汁！ </math>不得不说国产数据库为了让我们快速上手也是绞尽脑汁！

Oracle实时采集主要有两种实现方式， XStream API, or OpenLogReplicator

这里用OpenLogReplicator也就是LogMiner的方式，通过解析重做日志实现。

解析日志主要是这么几步

1. 通过当前的SCN号查询日志文件
2. 将日志文件搞进去解析，获得redo sql语句
3. 保存SCN号用于下次解析

所以我们参考Oracle的实现，讲解析日志的sql语句替换成达梦语法的sql语句即可。

注意：Oracle的SCN在达梦中为LSN

部分代码：

public static String build(OracleConnectorConfig connectorConfig, OracleDatabaseSchema schema) {
    final StringBuilder query = new StringBuilder(1024);
    query.append("SELECT SCN, SQL_REDO, OPERATION_CODE, TIMESTAMP, XID, CSF, TABLE_NAME, SEG_OWNER, OPERATION, ");
    query.append("USERNAME, ROW_ID, ROLL_BACK, RS_ID, STATUS, INFO, SSN, THREAD# ");
    query.append("FROM ").append(LOGMNR_CONTENTS_VIEW).append(" ");

    // These bind parameters will be bound when the query is executed by the caller.
    query.append("WHERE SCN > ? AND SCN <= ? ");

    // The connector currently requires a "database.pdb.name" configuration property when using CDB mode.
    // If this property is provided, build a predicate that will be used in later predicates.
    final String pdbName = connectorConfig.getPdbName();
    final String pdbPredicate;
    if (!Strings.isNullOrEmpty(pdbName)) {
        // This predicate is used later to explicitly restrict certain OPERATION_CODE and DDL events by the
        // PDB database name while allowing all START, COMMIT, MISSING_SCN, and ROLLBACK operations
        // regardless of where they originate, i.e. the PDB or CDB$ROOT.
        pdbPredicate = "SRC_CON_NAME = '" + pdbName + "'";
    }
    else {
        // No PDB configuration provided, no PDB predicate is necessary.
        pdbPredicate = null;
    }

    // Excluded schemas, if defined
    // This prevents things such as picking DDL for changes to LogMiner tables in SYSTEM tablespace
    // or picking up DML changes inside the SYS and SYSTEM tablespaces.
    final String excludedSchemas = resolveExcludedSchemaPredicate("SEG_OWNER");
    if (excludedSchemas.length() > 0) {
        query.append("AND ").append(excludedSchemas).append(' ');
    }

    query.append("AND (");

    // Always include START, COMMIT, MISSING_SCN, and ROLLBACK operations
    query.append("(OPERATION_CODE IN (6,7,34,36)");

    if (!schema.storeOnlyCapturedTables()) {
        // In this mode, the connector will always be fed DDL operations for all tables even if they
        // are not part of the inclusion/exclusion lists. We will pass the PDB predicate here to then
        // restrict DDL operations to only the PDB database if not null.
        query.append(" OR ").append(buildDdlPredicate(pdbPredicate)).append(" ");
        // Insert, Update, Delete, SelectLob, LobWrite, LobTrim, and LobErase
        if (connectorConfig.isLobEnabled()) {
            query.append(") OR (OPERATION_CODE IN (1,2,3,9,10,11,29) ");
        }
        else {
            // Only capture UNSUPPORTED operations (255) when LOB is disabled to avoid
            // the logging handler writing duplicate entries due to re-mining strategy
            query.append(") OR (OPERATION_CODE IN (1,2,3,255) ");
        }
        if (pdbPredicate != null) {
            // Restrict Insert, Update, Delete, and optionally SelectLob, LobWrite, LobTrim, and LobErase by PDB
            query.append("AND ").append(pdbPredicate).append(' ');
        }
    }
    else {
        query.append(") OR (");
        if (pdbPredicate != null) {
            // We specify the PDB predicate here because it applies to the OPERATION_CODE predicates but
            // also the DDL predicate that is to follow later due to predicate groups, effectively
            // restricting all DML operations and DDL changes to the PDB only.
            query.append(pdbPredicate).append(" AND ");
        }
        // Insert, Update, Delete, SelectLob, LobWrite, LobTrim, and LobErase
        if (connectorConfig.isLobEnabled()) {
            query.append("(OPERATION_CODE IN (1,2,3,9,10,11,29) ");
        }
        else {
            // Only capture UNSUPPORTED operations (255) when LOB is disabled to avoid
            // the logging handler writing duplicate entries due to re-mining strategy
            query.append("(OPERATION_CODE IN (1,2,3,255) ");
        }
        // In this mode, the connector will filter DDL operations based on the table inclusion/exclusion lists
        // We pass "null" to the DDL predicate because we will have added the predicate earlier as a part of
        // the outer predicate group to also be applied to OPERATION_CODE
        query.append("OR ").append(buildDdlPredicate(null)).append(") ");
    }

    // Always ignore the flush table
    query.append("AND TABLE_NAME != '").append(LogWriterFlushStrategy.LOGMNR_FLUSH_TABLE).append("' ");

    String schemaPredicate = buildSchemaPredicate(connectorConfig);
    if (!Strings.isNullOrEmpty(schemaPredicate)) {
        query.append("AND ").append(schemaPredicate).append(" ");
    }

    String tablePredicate = buildTablePredicate(connectorConfig);
    if (!Strings.isNullOrEmpty(tablePredicate)) {
        query.append("AND ").append(tablePredicate).append(" ");
    }

    query.append("))");

    return query.toString();
}

Jar包链接

kingbase: pan.baidu.com/s/1Ez7J_R5Q... 提取码: 3da2

dm: pan.baidu.com/s/1flapwI_d... 提取码: b9je

以上两个都是基于debezium1.9开发，需要源码的可以在评论区D我。