【实操记录】Oracle数据整库同步至Apache Doris

程裕强2023-10-07 18:49

本文是Oracle数据整库同步至Apache Doris实操记录，仅供参考

参考：https://cn.selectdb.com/blog/104

1、Oracle 配置

复制代码 
[root@node1 oracle]# pwd
/u01/app/oracle
[root@node1 oracle]# mkdir recovery_area
[root@node1 oracle]# chown -R oracle:dba recovery_area

SQL> conn sys/ as sysdba
输入口令: 
已连接。
SQL> alter system set db_recovery_file_dest_size = 2G;
系统已更改。
SQL> alter system set db_recovery_file_dest='/u01/app/oracle/recovery_area' scope=spfile;
系统已更改。
SQL>
复制代码 
SQL> shutdown immediate;
数据库已经关闭。
已经卸载数据库。
ORACLE 例程已经关闭。
SQL> startup mount;
ORACLE 例程已经启动。

Total System Global Area 1068937216 bytes
Fixed Size                  2233344 bytes
Variable Size             624954368 bytes
Database Buffers          436207616 bytes
Redo Buffers                5541888 bytes
数据库装载完毕。
SQL> alter database archivelog;

数据库已更改。

SQL> alter database open;

数据库已更改。

SQL> archive log list;
数据库日志模式            存档模式
自动存档             启用
存档终点            USE_DB_RECOVERY_FILE_DEST
最早的联机日志序列     1
下一个存档日志序列   2
当前日志序列           2
SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;

数据库已更改。

SQL>

2、Oracle 数据

复制代码 
SQL> CREATE USER admin IDENTIFIED BY 123;

用户已创建。

SQL> GRANT dba TO admin;

授权成功。

SQL> quit;

[root@node1 ~]# sqlplus admin/123

SQL*Plus: Release 11.2.0.2.0 Production on 星期三 6月 28 08:06:29 2023

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

连接到: 
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production
SQL> 

CREATE TABLE t_person(ID NUMBER(10),NAME VARCHAR2(128) ,PRIMARY KEY(ID));
INSERT INTO t_person VALUES (1, 'zhangsan');
INSERT INTO t_person VALUES (2, 'lisi');
INSERT INTO t_person VALUES (3, 'wangwu');
CREATE TABLE t_dept(ID NUMBER(10),NAME VARCHAR2(128) ,PRIMARY KEY(ID));
INSERT INTO t_dept VALUES (1, 'HR');
INSERT INTO t_dept VALUES (2, 'IT');
commit;

3、启动Doris

复制代码 
[root@node1 doris-2.0]# fe/bin/start_fe.sh --daemon
[root@node1 doris-2.0]# be/bin/start_be.sh --daemon
[root@node1 doris-2.0]# jps
16081 DorisBE
14723 DorisFE
16158 Jps
[root@node1 doris-2.0]#

下载相关jar，复制到flink的lib目录

复制代码 
[root@node1 flink-1.17.1]# ll lib
总用量 270772
-rw-r--r-- 1  501 games    196491 5月  19 06:56 flink-cep-1.17.1.jar
-rw-r--r-- 1  501 games    542620 5月  19 06:59 flink-connector-files-1.17.1.jar
-rw-r--r-- 1  501 games    102472 5月  19 07:02 flink-csv-1.17.1.jar
-rw-r--r-- 1  501 games 135975541 5月  19 07:13 flink-dist-1.17.1.jar
-rw-r--r-- 1 root root    8452171 6月  27 23:12 flink-doris-connector-1.17-1.4.0.jar
-rw-r--r-- 1  501 games    180248 5月  19 07:02 flink-json-1.17.1.jar
-rw-r--r-- 1  501 games  21043319 5月  19 07:12 flink-scala_2.12-1.17.1.jar
-rw-r--r-- 1 root root   28702667 6月  28 08:28 flink-sql-connector-oracle-cdc-2.4.1.jar
-rw-r--r-- 1  501 games  15407424 5月  19 07:13 flink-table-api-java-uber-1.17.1.jar
-rw-r--r-- 1  501 games  38191226 5月  19 07:08 flink-table-planner-loader-1.17.1.jar
-rw-r--r-- 1  501 games   3146210 5月  19 06:56 flink-table-runtime-1.17.1.jar
-rw-r--r-- 1  501 games    208006 5月  17 06:07 log4j-1.2-api-2.17.1.jar
-rw-r--r-- 1  501 games    301872 5月  17 06:07 log4j-api-2.17.1.jar
-rw-r--r-- 1  501 games   1790452 5月  17 06:07 log4j-core-2.17.1.jar
-rw-r--r-- 1  501 games     24279 5月  17 06:07 log4j-slf4j-impl-2.17.1.jar
[root@node1 flink-1.17.1]#

启动flink集群

复制代码 
[root@node1 flink-1.17.1]# bin/start-cluster.sh 
Starting cluster.
Starting standalonesession daemon on host node1.
Starting taskexecutor daemon on host node1.
[root@node1 flink-1.17.1]# jps
11755 StandaloneSessionClusterEntrypoint
12091 TaskManagerRunner
12190 Jps
[root@node1 flink-1.17.1]#

5、测试同步操作

复制代码 
[root@node1 flink-1.17.1]# bin/flink run \
>     -Dexecution.checkpointing.interval=10s \
>     -Dparallelism.default=1 \
>     -c org.apache.doris.flink.tools.cdc.CdcTools \
>     lib/flink-doris-connector-1.17-1.4.0.jar \
>     oracle-sync-database \
>     --database test \
>     --oracle-conf hostname=node1 \
>     --oracle-conf port=1521 \
>     --oracle-conf username=admin \
>     --oracle-conf password=admin123 \
>     --oracle-conf database-name=XE \
>     --oracle-conf schema-name=ADMIN \
>     --including-tables "t_.*" \
>     --sink-conf fenodes=node1:8030 \
>     --sink-conf username=root \
>     --sink-conf password=123456\
>     --sink-conf jdbc-url=jdbc:mysql://node1:9030 \
>     --sink-conf sink.label-prefix=label \
>     --table-conf replication_num=1

Unknown operation oracle-sync-database
[root@node1 flink-1.17.1]#

异常：Unknown operation oracle-sync-database

处理办法：

需要使用最新的 flink-doris-connector 包https://repository.apache.org/content/repositories/snapshots/org/apache/doris/

复制代码 
[root@node1 flink-1.17.1]# ll lib
-rw-r--r-- 1  501 games    196491 5月  19 06:56 flink-cep-1.17.1.jar
-rw-r--r-- 1  501 games    542620 5月  19 06:59 flink-connector-files-1.17.1.jar
-rw-r--r-- 1  501 games    102472 5月  19 07:02 flink-csv-1.17.1.jar
-rw-r--r-- 1  501 games 135975541 5月  19 07:13 flink-dist-1.17.1.jar
-rw-r--r-- 1 root root    8526699 9月  21 12:46 flink-doris-connector-1.17-1.5.0-20230915.034854-2.jar
-rw-r--r-- 1  501 games    180248 5月  19 07:02 flink-json-1.17.1.jar
-rw-r--r-- 1  501 games  21043319 5月  19 07:12 flink-scala_2.12-1.17.1.jar
-rw-r--r-- 1 root root   28702667 6月  28 08:28 flink-sql-connector-oracle-cdc-2.4.1.jar
-rw-r--r-- 1  501 games  15407424 5月  19 07:13 flink-table-api-java-uber-1.17.1.jar
-rw-r--r-- 1  501 games  38191226 5月  19 07:08 flink-table-planner-loader-1.17.1.jar
-rw-r--r-- 1  501 games   3146210 5月  19 06:56 flink-table-runtime-1.17.1.jar
-rw-r--r-- 1  501 games    208006 5月  17 06:07 log4j-1.2-api-2.17.1.jar
-rw-r--r-- 1  501 games    301872 5月  17 06:07 log4j-api-2.17.1.jar
-rw-r--r-- 1  501 games   1790452 5月  17 06:07 log4j-core-2.17.1.jar
-rw-r--r-- 1  501 games     24279 5月  17 06:07 log4j-slf4j-impl-2.17.1.jar

重新执行

复制代码 
[root@node1 flink-1.17.1]# bin/flink run \
>     -Dexecution.checkpointing.interval=10s \
>     -Dparallelism.default=1 \
>     -c org.apache.doris.flink.tools.cdc.CdcTools \
>     lib/flink-doris-connector-1.17-1.5.0-20230915.034854-2.jar \
>     oracle-sync-database \
>     --database test \
>     --oracle-conf hostname=node1 \
>     --oracle-conf port=1521 \
>     --oracle-conf username=admin \
>     --oracle-conf password=admin123 \
>     --oracle-conf database-name=XE \
>     --oracle-conf schema-name=ADMIN \
>     --including-tables "t_.*" \
>     --sink-conf fenodes=node1:8030 \
>     --sink-conf username=root \
>     --sink-conf password=123456\
>     --sink-conf jdbc-url=jdbc:mysql://node1:9030 \
>     --sink-conf sink.label-prefix=label \
>     --table-conf replication_num=1


------------------------------------------------------------
 The program finished with the following exception:

org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: No tables to be synchronized.
        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
        at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
        at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105)
        at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:851)
        at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:245)
        at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1095)
        at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$9(CliFrontend.java:1189)
        at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
        at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1189)
        at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1157)
Caused by: java.lang.IllegalStateException: No tables to be synchronized.
        at org.apache.flink.util.Preconditions.checkState(Preconditions.java:193)
        at org.apache.doris.flink.tools.cdc.DatabaseSync.build(DatabaseSync.java:101)
        at org.apache.doris.flink.tools.cdc.CdcTools.syncDatabase(CdcTools.java:116)
        at org.apache.doris.flink.tools.cdc.CdcTools.createOracleSyncDatabase(CdcTools.java:80)
        at org.apache.doris.flink.tools.cdc.CdcTools.main(CdcTools.java:53)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
        ... 9 more
[root@node1 flink-1.17.1]#

报错：No tables to be synchronized

修改：--including-tables "T_.*"

复制代码 
[root@node1 flink-1.17.1]# bin/flink run \
>     -Dexecution.checkpointing.interval=10s \
>     -Dparallelism.default=1 \
>     -c org.apache.doris.flink.tools.cdc.CdcTools \
>     lib/flink-doris-connector-1.17-1.5.0-20230915.034854-2.jar \
>     oracle-sync-database \
>     --database test \
>     --oracle-conf hostname=node1 \
>     --oracle-conf port=1521 \
>     --oracle-conf username=admin \
>     --oracle-conf password=admin123 \
>     --oracle-conf database-name=XE \
>     --oracle-conf schema-name=ADMIN \
>     --including-tables "T_.*" \
>     --sink-conf fenodes=node1:8030 \
>     --sink-conf username=root \
>     --sink-conf password=123456\
>     --sink-conf jdbc-url=jdbc:mysql://node1:9030 \
>     --sink-conf sink.label-prefix=label \
>     --table-conf replication_num=1


------------------------------------------------------------
 The program finished with the following exception:

org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: The following SQL query could not be executed: SELECT `SCHEMA_NAME` FROM `INFORMATION_SCHEMA`.`SCHEMATA`;
        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
        at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
        at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105)
        at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:851)
        at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:245)
        at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1095)
        at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$9(CliFrontend.java:1189)
        at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
        at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1189)
        at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1157)
Caused by: org.apache.doris.flink.exception.DorisSystemException: The following SQL query could not be executed: SELECT `SCHEMA_NAME` FROM `INFORMATION_SCHEMA`.`SCHEMATA`;
        at org.apache.doris.flink.catalog.doris.DorisSystem.extractColumnValuesBySQL(DorisSystem.java:137)
        at org.apache.doris.flink.catalog.doris.DorisSystem.listDatabases(DorisSystem.java:59)
        at org.apache.doris.flink.catalog.doris.DorisSystem.databaseExists(DorisSystem.java:67)
        at org.apache.doris.flink.tools.cdc.DatabaseSync.build(DatabaseSync.java:102)
        at org.apache.doris.flink.tools.cdc.CdcTools.syncDatabase(CdcTools.java:116)
        at org.apache.doris.flink.tools.cdc.CdcTools.createOracleSyncDatabase(CdcTools.java:80)
        at org.apache.doris.flink.tools.cdc.CdcTools.main(CdcTools.java:53)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
        ... 9 more
Caused by: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:67)
        at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74)
        at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at org.apache.doris.flink.connection.SimpleJdbcConnectionProvider.getOrEstablishConnection(SimpleJdbcConnectionProvider.java:52)
        at org.apache.doris.flink.catalog.doris.DorisSystem.extractColumnValuesBySQL(DorisSystem.java:121)
        ... 20 more
[root@node1 flink-1.17.1]#

缺少mysql驱动，flink的lib目录添加mysql-connector-java-5.1.49.jar

复制代码 
[root@node1 flink-1.17.1]# bin/flink run \
>     -Dexecution.checkpointing.interval=10s \
>     -Dparallelism.default=1 \
>     -c org.apache.doris.flink.tools.cdc.CdcTools \
>     lib/flink-doris-connector-1.17-1.5.0-20230915.034854-2.jar \
>     oracle-sync-database \
>     --database test \
>     --oracle-conf hostname=node1 \
>     --oracle-conf port=1521 \
>     --oracle-conf username=admin \
>     --oracle-conf password=admin123 \
>     --oracle-conf database-name=XE \
>     --oracle-conf schema-name=ADMIN \
>     --including-tables "T_.*" \
>     --sink-conf fenodes=node1:8030 \
>     --sink-conf username=root \
>     --sink-conf password=123456\
>     --sink-conf jdbc-url=jdbc:mysql://node1:9030 \
>     --sink-conf sink.label-prefix=label \
>     --table-conf replication_num=1

Thu Sep 21 13:06:59 EDT 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Job has been submitted with JobID fd244bcb10f06e5aa801d2708441bb86

登录Doris查看同步效果

复制代码 
[root@node1 doris-2.0]# mysql -h192.168.203.101 -P9030 -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.99 Doris version doris-2.0.1.1-ea0bfb2

Copyright (c) 2000, 2021, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| T_DEPT         |
| T_PERSON       |
+----------------+
2 rows in set (0.00 sec)

mysql> select * from T_PERSON;
Empty set (0.29 sec)

mysql> select * from T_DEPT;
Empty set (0.06 sec)

6、同步成功

复制代码 
bin/flink run \
    -Dexecution.checkpointing.interval=10s \
    -Dparallelism.default=1 \
    -c org.apache.doris.flink.tools.cdc.CdcTools \
    lib/flink-doris-connector-1.17-1.5.0-20230915.034854-2.jar \
    oracle-sync-database \
    --database test \
    --oracle-conf hostname=node1 \
    --oracle-conf port=1521 \
    --oracle-conf username=admin \
    --oracle-conf password=123 \
    --oracle-conf database-name=XE \
    --oracle-conf schema-name=ADMIN \
    --including-tables "T_.*" \
    --sink-conf fenodes=node1:8030 \
    --sink-conf username=root \
    --sink-conf password=123456\
    --sink-conf jdbc-url=jdbc:mysql://node1:9030 \
    --sink-conf sink.label-prefix=label \
    --table-conf replication_num=1

查看Flink中运行的Job http://192.168.203.101:8081/

7、测试新增记录同步

复制代码 
SQL> INSERT INTO t_person VALUES (4, 'test_user');

已创建 1 行。

SQL> INSERT INTO t_dept VALUES (3, 'CS');

已创建 1 行。

SQL> commit;

提交完成。

SQL>
相关推荐
java_logo2 小时前
Apache Doris Docker 部署指南：实时分析数据库实战
数据库·docker·apache·doris·apache doris·轩辕镜像·docker部署doris
Cloud云卷云舒11 小时前
云卷云舒：从Oracle/MySQL迁移到HaishanDB：迁移评估工具核心技术拆解
mysql·oracle·ai-native·haishandb·信创数据库替换
段一凡-华北理工大学12 小时前
向量数据库实战：选型、调优与落地~系列文章12：文本分块策略实战：chunk_size 怎么选？重叠多少？
开发语言·数据库·后端·oracle·rust·工业智能体·高炉智能化
oradh14 小时前
Oracle XTTS实现跨版本迁移和升级（Oracle 11g单库升级至19C RAC集群）
数据库·oracle·11g升级19c·xtts跨版本迁移和升级
会编程的土豆17 小时前
数据库范式
数据库·oracle
—Miss. Z—19 小时前
计算机二级MySQL——简单应用题（存储过程&存储函数）
数据库·oracle·php
翔云12345620 小时前
Kafka + Flink实时时流处理场景
flink·kafka
NineData1 天前
Oracle 卡住时先看阻塞源，ChatDBA 会先把锁链路理出来
数据库·oracle·ninedata·锁故障·锁等待·chatdba·锁阻塞
踏月的造梦星球2 天前
达梦数据库执行计划与性能分析入门：EXPLAIN、AUTOTRACE 与 ET
服务器·数据库·oracle
尽兴-2 天前
企业数据库选型与演进：Oracle、达梦、MySQL 与分库分表实践
数据库·mysql·oracle·分库分表
热门推荐
012026年7月AI圈大地震：GPT-5.6被政府限制、Claude入驻Slack、Anthropic自研芯片02GitHub 镜像站点03如何新建文件夹？ 电脑新建文件夹的4种方法04AI科技热点日报 | 2026年07月01日05幻兽帕鲁 - 服务器管理员权限与 GM 命令完全指南06国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）072026 国产 AI 大模型横评：DeepSeek、通义千问、Kimi、文心一言、星火、豆包谁更能打？08AI 编程 IDE 全景解析 2026：Agent 全面接管开发链路09微信历史版本含下载地址（ Windows PC | 安卓 | MAC ）及设置微信不更新102026 年 AI 大模型 & AI 编程工具实战全总结