datax离线同步oracle表到clickhouse实践1

时间：2024.01

目录
1、安装启动 oracle19c 容器
2、rpm包安装clickhouse
3、datax安装

4、datax同步

目标库根据要同步的表，按照clickhouse建表规范建表

编写json文件

编写增量同步shell脚本，加入 crond 定时任务

1、安装启动 oracle19c 容器

二、拉取镜像

docker pull registry.cn-hangzhou.aliyuncs.com/zhuyijun/oracle

三. 创建Oracle 19c镜像的容器

3.1创建目录

root@docker \~\]# mkdir /opt/oradata \[root@docker \~\]# chmod -R 777 /opt/oradata/ \[root@docker \~\]# mkdir /opt/to_oracle 3.2 创建实例 docker run -d -it --name oracle19c -p 11521:1521 -p 5500:5500 \\ -v /opt/oradata:/opt/oracle/oradata \\ -v /opt/to_oracle:/home/oracle/to_oracle \\ registry.cn-hangzhou.aliyuncs.com/zhuyijun/oracle:latest 3.3 查看日志 docker logs -f oracle19c 四、 修改密码 # 系统默认的system密码修改成oracle docker exec oracle19c ./setPassword.sh oracle 5.1 进入容器 docker exec -it oracle19c /bin/bash 5.2 登录数据库 sqlplus sys/oracle@localhost:1521/ORCLCDB as sysdba show pdbs; alter session set container=ORCLPDB1; 5.4 查看数据文件位置 select file_name from dba_data_files; 5.6 创建数据表空间 create tablespace bigdata datafile '/opt/oracle/oradata/ORCLCDB/ORCLPDB1/bigdata.dbf' size 10G autoextend on; 5.7 创建用户，指定默认表空间 create user bigdata identified by bigdata default tablespace bigdata; 5.8 用户授权 grant connect,resource,dba to bigdata; 5.3 连接可拔插数据库 sqlplus /nolog conn bigdata/bigdata@ORCLPDB1 建表、插入数据 把 create.sql insert.sql 放在 /opt/to_oracle/ SQL\> @/home/oracle/to_oracle/create.sql 文件内容 create table TDBA_TEST01 ( TID number primary key, TSN varchar2(40), TNO number, TAMT number(15,2), CREATE_DATE date not null, UPDATE_DATE date, UPDATE_TIME date ); SQL\> @/home/oracle/to_oracle/insert.sql 文件内容 INSERT INTO TDBA_TEST01 VALUES(1,'SN001',1,1.01,TO_DATE('2020-02-01','YYYY-MM-DD'),TO_DATE('2020-02-01','YYYY-MM-DD'),TO_DATE('2020-02-01 01:01:01','YYYY-MM-DD HH24:MI:SS')); INSERT INTO TDBA_TEST01 VALUES(2,'SN002',2,2.12,TO_DATE('2020-02-02','YYYY-MM-DD'),TO_DATE('2020-02-02','YYYY-MM-DD'),TO_DATE('2020-02-02 01:01:01','YYYY-MM-DD HH24:MI:SS')); INSERT INTO TDBA_TEST01 VALUES(3,'SN003',3,3.23,TO_DATE('2020-02-03','YYYY-MM-DD'),TO_DATE('2020-02-03','YYYY-MM-DD'),TO_DATE('2020-02-03 01:01:01','YYYY-MM-DD HH24:MI:SS')); INSERT INTO TDBA_TEST01 VALUES(4,'SN004',4,4.34,TO_DATE('2020-02-04','YYYY-MM-DD'),TO_DATE('2020-02-04','YYYY-MM-DD'),TO_DATE('2020-02-04 01:01:01','YYYY-MM-DD HH24:MI:SS')); INSERT INTO TDBA_TEST01 VALUES(5,'SN005',5,5.45,TO_DATE('2020-02-05','YYYY-MM-DD'),TO_DATE('2020-02-05','YYYY-MM-DD'),TO_DATE('2020-02-05 01:01:01','YYYY-MM-DD HH24:MI:SS')); INSERT INTO TDBA_TEST01 VALUES(6,'SN006',6,6,TO_DATE('2020-02-06','YYYY-MM-DD'),TO_DATE('2020-02-06','YYYY-MM-DD'),TO_DATE('2020-02-06 01:01:01','YYYY-MM-DD HH24:MI:SS')); INSERT INTO TDBA_TEST01 VALUES(7,'SN007',7,7.67,TO_DATE('2020-02-07','YYYY-MM-DD'),TO_DATE('2020-02-07','YYYY-MM-DD'),TO_DATE('2020-02-07 01:01:01','YYYY-MM-DD HH24:MI:SS')); INSERT INTO TDBA_TEST01 VALUES(8,'SN008',8,8.78,TO_DATE('2020-02-08','YYYY-MM-DD'),TO_DATE('2020-02-08','YYYY-MM-DD'),TO_DATE('2020-02-08 01:01:01','YYYY-MM-DD HH24:MI:SS')); INSERT INTO TDBA_TEST01 VALUES(9,'SN009',9,9,TO_DATE('2020-02-09','YYYY-MM-DD'),TO_DATE('2024-01-09','YYYY-MM-DD'),TO_DATE('2020-02-09 01:01:01','YYYY-MM-DD HH24:MI:SS')); INSERT INTO TDBA_TEST01 VALUES(9,'SN009',9,9,TO_DATE('2020-02-09','YYYY-MM-DD'),TO_DATE('2024-01-09','YYYY-MM-DD'),TO_DATE('2020-02-09 01:01:01','YYYY-MM-DD HH24:MI:SS')); **2、rpm包安装clickhouse** \[root@conda \~\]# mkdir clickhouse22.3 \[root@conda \~\]# cd clickhouse22.3 下载clickhouse的rpm包 https://packages.clickhouse.com/rpm/stable/clickhouse-client-22.3.2.2-2.noarch.rpm https://packages.clickhouse.com/rpm/stable/clickhouse-common-static-22.3.2.2-2.x86_64.rpm https://packages.clickhouse.com/rpm/stable/clickhouse-common-static-dbg-22.3.2.2-2.x86_64.rpm https://packages.clickhouse.com/rpm/stable/clickhouse-server-22.3.2.2-2.noarch.rpm \[root@conda clickhouse22.3\]# ll total 929924 -rw-r--r--. 1 root root 46748 Sep 22 2022 clickhouse-client-22.3.2.2-2.noarch.rpm -rw-r--r--. 1 root root 178362876 Sep 22 2022 clickhouse-common-static-22.3.2.2-2.x86_64.rpm -rw-r--r--. 1 root root 773751886 Sep 22 2022 clickhouse-common-static-dbg-22.3.2.2-2.x86_64.rpm -rw-r--r--. 1 root root 72089 Sep 22 2022 clickhouse-server-22.3.2.2-2.noarch.rpm \[root@conda clickhouse22.3\]# rpm -ivh \*.rpm ![](https://file.jishuzhan.net/article/1757058205667561473/d7a537a1eb38d46a1a429cb1e8327aac.webp) 自定义设置密码 bigdata ![](https://file.jishuzhan.net/article/1757058205667561473/ec34e0296e6a0dd4bbc309baff0d6cc3.webp) \[root@conda clickhouse22.3\]# clickhouse start \[root@conda clickhouse22.3\]# clickhouse status /var/run/clickhouse-server/clickhouse-server.pid file exists and contains pid = 10792. The process with pid = 10792 is running. \[root@conda clickhouse22.3\]# clickhouse-client --password bigdata -m ClickHouse client version 22.3.2.1. Connecting to localhost:9000 as user default. Connected to ClickHouse server version 22.3.2 revision 54455. conda :) create table TEST01 ( TID UInt32, TSN String, TNO UInt16, TAMT Decimal(15,2), CREATE_DATE Date, UPDATE_DATE Date, UPDATE_TIME DateTime ) ENGINE = MergeTree(CREATE_DATE, (TID), 8192) **3、datax安装** System Requirements JDK(1.8以上，推荐1.8) Python(推荐Python2.6.X) 我使用的CentOS7自带的 Python 2.7.5 直接下载DataX工具包： DataX下载地址（http://datax-opensource.oss-cn-hangzhou.aliyuncs.com/datax.tar.gz） datax_ck.tar.gz [datax_ck.tar.gz下载![icon-default.png?t=N7T8](https://file.jishuzhan.net/article/1757058205667561473/c9868f5524aa31bbc2ae673827f18730.webp)https://www.123pan.com/s/VTMXjv-pfA6v.html](https://www.123pan.com/s/VTMXjv-pfA6v.html "datax_ck.tar.gz下载") 提取码:xxxx 下载后解压至本地某个目录，进入bin目录，即可运行同步作业： tar -zxvf datax_ck.tar.gz