Apache SeaTunnel配置使用案例

前置操作

Apache SeaTunnel详解与部署（最新版本2.3.11）-CSDN博客

mkdir /usr/local/soft/apache-seatunnel-2.3.11/job/

一、MySQL to HDFS

官方配置参考：

MySQL | Apache SeaTunnel

Hdfs文件 | Apache SeaTunnel

1、配置确认

将mysql-connector-java-8.0.15.jar放入到/usr/local/soft/apache-seatunnel-2.3.11/lib/文件夹中

2、启动Hadoop

3、MySQL数据库表创建

确认MySQL数据库wjobs表user中有数据，没有则进行创建

4、创建seatunnel配置文件

cd /usr/local/soft/apache-seatunnel-2.3.11/job/
touch mysql2hdfs.conf

内容如下：

env {
  parallelism = 2
  job.mode = "BATCH"
}
source {
    Jdbc {
        url = "jdbc:mysql://node11:3306/wjobs"
        driver = "com.mysql.cj.jdbc.Driver"
        connection_check_timeout_sec = 100
        user = "root"
        password = "root123"
        table_path = "wjobs.user"
        query = "select  * from wjobs.user"
    }
}

sink {
  HdfsFile {
    fs.defaultFS = "hdfs://node11:9000"
    path = "/seatunnel_data/user"
    file_format_type = "text"
    field_delimiter = "\t"
    row_delimiter = "\n"
    custom_filename = true
    file_name_expression = "${transactionId}_${now}"
    filename_time_format = "yyyy.MM.dd"
    sink_columns = ["id","username","password"]
    is_enable_transaction = true
}
}

或

env {
  parallelism = 2
  job.mode = "BATCH"
}
source {
    Jdbc {
        url = "jdbc:mysql://node11:3306/wjobs"
        driver = "com.mysql.cj.jdbc.Driver"
        connection_check_timeout_sec = 100
        user = "root"
        password = "root123"
        table_path = "wjobs.user"
        query = "select  * from wjobs.user"
    }
}

sink {
  HdfsFile {
    fs.defaultFS = "hdfs://node11:9000"
    path = "/seatunnel_data/user"
    file_format_type = "text"
    field_delimiter = "\t"
    row_delimiter = "\n"
    is_enable_transaction = true
}
}

5、执行seatunnel转换命令

cd /usr/local/soft/apache-seatunnel-2.3.11/
./bin/seatunnel.sh --config ./job/mysql2hdfs.conf -m local

注意：此为单次启动，不用启动seatunnel服务

二、Hive to MySQL

参考：Hive | Apache SeaTunnel

MySQL | Apache SeaTunnel

1、准备数据

确认hive中表及数据、MySQL对应表已创建

2、创建seatunnel配置文件

cd /usr/local/soft/apache-seatunnel-2.3.11/job/
touch hive2mysql.conf

内容如下：

env {
  parallelism = 2
  job.mode = "BATCH"
}
source {
  Hive {
    table_name = "uoapdb.ext_startup_logs"
    metastore_uri = "thrift://node11:9083"
  }
}

sink {
  jdbc {
        url = "jdbc:mysql://node11:3306/wjobs?useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"
        driver = "com.mysql.cj.jdbc.Driver"
        user = "root"
        password = "root123"
        query = "insert into logs(userId,platform) values(?,?)"
        }
}

3、 启动Hive

nohup hive --service metastore &
nohup hiveserver2 &

4、执行命令

cd /usr/local/soft/apache-seatunnel-2.3.11/
./bin/seatunnel.sh --config ./job/mysql2hdfs.conf -m local