Debezium
Debezium: 一款国外开源的数据库实时捕获项目，目前最新版本为2023-10-10的2.4，支持主流的数据库包括：
有两种部署模式，包括独立部署（server+kafka）以及嵌入式部署（内嵌到项目中）。
如何兼容人大金仓
通过考古国外人大金仓官网后发现该国产数据库是基于PostgreSQL研发的，于是试试直接用PostgreSQL的连接器是否可以运行。由于kingbase沿用了pg的复制槽，所以我们主要关注解码插件好了。 三种插件：
- pgoutput
- pgproto
- testdecoding
| 插件名 | 是否自带 | 可行性 | 安装方式 |
|---|---|---|---|
| pgoutput | 否 | 可行 | 编译部署 |
| pgproto | 否 | 由于无法获得.proto协议文件，可行性不大 | 无需安装 |
| testdecoding | 是 | 可行 | 无需安装 |
这里采用testdecoding插件，只需增加testdecoding解析代码即可，其他配置基本参照pg。
csharp
public enum MessageType {
RELATION,
BEGIN,
COMMIT,
INSERT,
UPDATE,
DELETE;
public static MessageType forType(String context) {
List<String> strings;
if (Strings.isNullOrEmpty(context) || isEmpty(strings = Splitter.on(" ").splitToList(context))) {
throw new InvalidParameterException("Invalid message type");
}
String first = strings.get(0);
if ("BEGIN".equals(first)) {
return BEGIN;
}
else if ("COMMIT".equals(first)) {
return COMMIT;
}
switch (strings.get(2)) {
case "INSERT:":
return INSERT;
case "UPDATE:":
return UPDATE;
case "DELETE:":
return DELETE;
default:
return RELATION;
}
}
}
public TestDecodingMessageDecoder(MessageDecoderContext decoderContext, PostgresConnection connection) {
this.decoderContext = decoderContext;
this.connection = connection;
}
如何兼容达梦
通过挖掘国内达梦官网后发现其基本基于oracle研发，许多概念都与oracle相似，惊觉走kingbase的套路或许可行。 不得不说国产数据库为了让我们快速上手也是绞尽脑汁！
Oracle实时采集主要有两种实现方式， XStream API, or OpenLogReplicator
这里用OpenLogReplicator也就是LogMiner的方式，通过解析重做日志实现。
解析日志主要是这么几步
- 通过当前的SCN号查询日志文件
- 将日志文件搞进去解析，获得redo sql语句
- 保存SCN号用于下次解析
所以我们参考Oracle的实现，讲解析日志的sql语句替换成达梦语法的sql语句即可。
注意：Oracle的SCN在达梦中为LSN
部分代码：
scss
public static String build(OracleConnectorConfig connectorConfig, OracleDatabaseSchema schema) {
final StringBuilder query = new StringBuilder(1024);
query.append("SELECT SCN, SQL_REDO, OPERATION_CODE, TIMESTAMP, XID, CSF, TABLE_NAME, SEG_OWNER, OPERATION, ");
query.append("USERNAME, ROW_ID, ROLL_BACK, RS_ID, STATUS, INFO, SSN, THREAD# ");
query.append("FROM ").append(LOGMNR_CONTENTS_VIEW).append(" ");
// These bind parameters will be bound when the query is executed by the caller.
query.append("WHERE SCN > ? AND SCN <= ? ");
// The connector currently requires a "database.pdb.name" configuration property when using CDB mode.
// If this property is provided, build a predicate that will be used in later predicates.
final String pdbName = connectorConfig.getPdbName();
final String pdbPredicate;
if (!Strings.isNullOrEmpty(pdbName)) {
// This predicate is used later to explicitly restrict certain OPERATION_CODE and DDL events by the
// PDB database name while allowing all START, COMMIT, MISSING_SCN, and ROLLBACK operations
// regardless of where they originate, i.e. the PDB or CDB$ROOT.
pdbPredicate = "SRC_CON_NAME = '" + pdbName + "'";
}
else {
// No PDB configuration provided, no PDB predicate is necessary.
pdbPredicate = null;
}
// Excluded schemas, if defined
// This prevents things such as picking DDL for changes to LogMiner tables in SYSTEM tablespace
// or picking up DML changes inside the SYS and SYSTEM tablespaces.
final String excludedSchemas = resolveExcludedSchemaPredicate("SEG_OWNER");
if (excludedSchemas.length() > 0) {
query.append("AND ").append(excludedSchemas).append(' ');
}
query.append("AND (");
// Always include START, COMMIT, MISSING_SCN, and ROLLBACK operations
query.append("(OPERATION_CODE IN (6,7,34,36)");
if (!schema.storeOnlyCapturedTables()) {
// In this mode, the connector will always be fed DDL operations for all tables even if they
// are not part of the inclusion/exclusion lists. We will pass the PDB predicate here to then
// restrict DDL operations to only the PDB database if not null.
query.append(" OR ").append(buildDdlPredicate(pdbPredicate)).append(" ");
// Insert, Update, Delete, SelectLob, LobWrite, LobTrim, and LobErase
if (connectorConfig.isLobEnabled()) {
query.append(") OR (OPERATION_CODE IN (1,2,3,9,10,11,29) ");
}
else {
// Only capture UNSUPPORTED operations (255) when LOB is disabled to avoid
// the logging handler writing duplicate entries due to re-mining strategy
query.append(") OR (OPERATION_CODE IN (1,2,3,255) ");
}
if (pdbPredicate != null) {
// Restrict Insert, Update, Delete, and optionally SelectLob, LobWrite, LobTrim, and LobErase by PDB
query.append("AND ").append(pdbPredicate).append(' ');
}
}
else {
query.append(") OR (");
if (pdbPredicate != null) {
// We specify the PDB predicate here because it applies to the OPERATION_CODE predicates but
// also the DDL predicate that is to follow later due to predicate groups, effectively
// restricting all DML operations and DDL changes to the PDB only.
query.append(pdbPredicate).append(" AND ");
}
// Insert, Update, Delete, SelectLob, LobWrite, LobTrim, and LobErase
if (connectorConfig.isLobEnabled()) {
query.append("(OPERATION_CODE IN (1,2,3,9,10,11,29) ");
}
else {
// Only capture UNSUPPORTED operations (255) when LOB is disabled to avoid
// the logging handler writing duplicate entries due to re-mining strategy
query.append("(OPERATION_CODE IN (1,2,3,255) ");
}
// In this mode, the connector will filter DDL operations based on the table inclusion/exclusion lists
// We pass "null" to the DDL predicate because we will have added the predicate earlier as a part of
// the outer predicate group to also be applied to OPERATION_CODE
query.append("OR ").append(buildDdlPredicate(null)).append(") ");
}
// Always ignore the flush table
query.append("AND TABLE_NAME != '").append(LogWriterFlushStrategy.LOGMNR_FLUSH_TABLE).append("' ");
String schemaPredicate = buildSchemaPredicate(connectorConfig);
if (!Strings.isNullOrEmpty(schemaPredicate)) {
query.append("AND ").append(schemaPredicate).append(" ");
}
String tablePredicate = buildTablePredicate(connectorConfig);
if (!Strings.isNullOrEmpty(tablePredicate)) {
query.append("AND ").append(tablePredicate).append(" ");
}
query.append("))");
return query.toString();
}Jar包链接
kingbase: pan.baidu.com/s/1Ez7J_R5Q... 提取码: 3da2
dm: pan.baidu.com/s/1flapwI_d... 提取码: b9je
以上两个都是基于debezium1.9开发，需要源码的可以在评论区D我。