使用阿里云DataX完成数据同步

千月落2023-08-05 11:17

DataX

DataX 是阿里云 DataWorks 数据集成的开源版本，在阿里巴巴集团内被广泛使用的离线数据同步工具/平台。DataX 实现了包括 MySQL、Oracle、OceanBase、SqlServer、Postgre、HDFS、Hive、ADS、HBase、TableStore(OTS)、MaxCompute(ODPS)、Hologres、DRDS, databend 等各种异构数据源之间高效的数据同步功能。

官网：https://github.com/alibaba/DataX

环境准备

参考官网：https://github.com/alibaba/DataX/blob/master/userGuid.md

下载解压后

MongoDB同步数据至达梦数据库

新建xxx.json，写入如下内容

javascript 复制代码

  {
  "job": {
      "setting": {
          "speed": {
              "channel": 10
          }
      },
      "content": [
          {
              "reader": {
                  "name": "mongodbreader",
                  "parameter": {
                      "address": ["192.168.2.39:27017"],
                      "userName": "",
                      "userPassword": "",
                      "dbName": "ztcx_test",
                      "collectionName": "gps_driver_location_20230718",
                      "column": [
                          {
                              "name": "longitude",
                              "type": "Double"
                          },
                          {
                              "name": "latitude",
                              "type": "Double"
                          },
                          {
                              "name": "location_type",
                              "type": "int"
                          },
                          {
                              "name": "type",
                              "type": "int"
                          },
                          {
                              "name": "driver_id",
                              "type": "Long"
                          },
                          {
                              "name": "plate_no",
                              "type": "string"
                          },
                          {
                              "name": "create_date",
                              "type": "Date",
                          }
                      ]
                  }
              },
              "writer": {
                  "name": "rdbmswriter",
                  "parameter": {
                     "connection": [
                            {
                                "jdbcUrl": "jdbc:dm://192.168.2.27:5236/DATAX",
                                "table": [
                                    "DRIVER_LOCATION"
                                ]
                            }
                        ],
                        "username": "xxx",
                        "password": "****",
                        "table": "DRIVER_LOCATION",
                        "column": [
                           "longitude",
						   "latitude",
						   "location_type",
						   "type",
						   "driver_id",
						   "plate_no",
						   "create_date"
                        ],
                        "preSql": [
                            "delete from DRIVER_LOCATION;"
                        ]
                  }
              }
          }
      ]
  }
  }

配置文件属性，可参考

切换至bin目录，执行以下命令即可

bash 复制代码

python datax.py ./xxx.json