Seatunnel Docker image镜像制作

Seatunnel Docker image镜像制作

    #下载 seatunnel

    • export version="2.3.3"
    • wget "Index of /dist/seatunnel{version}/apache-seatunnel-{version}-bin.tar.gz"
    • tar -xzvf "apache-seatunnel-${version}-bin.tar.gz"

    #解压

    • tar -xzvf apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz

    #配置connector, config/plugin_config 根据需要配置,Demo只需要以下两个即可

    • --seatunnel-connectors--
    • connector-fake
    • connector-console
    • --end--

    #执行插件安装

    • sh bin/install-plugin.sh 2.3.3
    • #执行后,会自动下载maven包到 ~/.m2/wrapper/dists/apache-maven-3.8.4-bin/目录中
    • #连接到 Central Repository:,下载传统比较慢
    • #需要一下镜像,全用本地的setting.xml
      • #:~/.m2/wrapper/dists/apache-maven-3.8.4-bin/52ccbt68d252mdldqsfsn03jlf/apache-maven-3.8.4/conf#
    • #再重新执行 sh bin/install-plugin.sh 2.3.3 使用国内镜像,例如:阿里云,此时就很快了
    • #bin/install-plugin.sh 会将对就的jar包复制到 connectors/seatunnel和lib目录下

    #修改apache-seatunnel-${version}-bin目录下的配置文件

    • #编写plugin_config文件
      • vi config/plugin_config
        • --seatunnel-connectors--
        • connector-fake
        • connector-console
        • --end--
    • #编写批处理配置文件
      • vi config/v2.batch.config.template

    env {
    execution.parallelism = 1
    job.mode = "BATCH"
    }

    source {
    FakeSource {
    result_table_name = "fake"
    row.num = 16
    schema = {
    fields {
    name = "string"
    age = "int"
    }
    }
    }
    }

    transform {
    FieldMapper {
    source_table_name = "fake"
    result_table_name = "fake1"
    field_mapper = {
    age = age
    name = new_name
    }
    }
    }

    sink {
    Console {
    source_table_name = "fake1"
    }
    }

    #下载openjdk:8镜像

    • docker pull openjdk:8

    #创建Dockerfile

    • vi dockerfile-seatunnel-2.3.3
    • #内容 当前目录下的seatunnel包、lib目录、connectors目录复制到镜像中
      • FROM openjdk:8
      • ENV SEATUNNEL_VERSION="2.3.3"
      • COPY ./apache-seatunnel-{SEATUNNEL_VERSION}-bin.tar.gz /opt/apache-seatunnel-{SEATUNNEL_VERSION}-bin.tar.gz
      • WORKDIR /opt
      • RUN tar -xzvf apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
      • RUN mv apache-seatunnel-${SEATUNNEL_VERSION} seatunnel
      • RUN rm -f /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
      • WORKDIR /opt/seatunnel
      • ENTRYPOINT ["sh","-c"," bin/seatunnel.sh --config $config -e local"]

    #build镜像

    • docker build -t seatunnel:2.3.3 -f dockerfile-seatunnel-2.3.3 .

    #使用镜像后台运行

    • docker run -d -p 9000:9000 --restart=unless-stopped --name seatunnel -d --hostname seatunnel-node1 --network my-net -e config="/data/seatunnel.batch.conf" -v /mnt/sda1/seatunnel/apache-seatunnel-2.3.3/config/v2.batch.config.template:/data/seatunnel.batch.conf -v /mnt/sda1/seatunnel/apache-seatunnel-2.3.3/config/plugin_config:/opt/seatunnel/plugin_config -v /mnt/sda1/seatunnel/apache-seatunnel-2.3.3/lib:/opt/seatunnel/lib -v /mnt/sda1/seatunnel/apache-seatunnel-2.3.3/plugins:/opt/seatunnel/plugins -v /mnt/sda1/seatunnel/apache-seatunnel-2.3.3/connectors/seatunnel:/opt/seatunnel/connectors/seatunnel -v /etc/localtime:/etc/localtime seatunnel:2.3.3

    #使用镜像临时测试

    • docker run --name seatunnel --hostname seatunnel-node1 --network my-net -e config="/data/seatunnel.batch.conf" -v /mnt/sda1/seatunnel/apache-seatunnel-2.3.3/config/v2.batch.config.template:/data/seatunnel.batch.conf -v /mnt/sda1/seatunnel/apache-seatunnel-2.3.3/config/plugin_config:/opt/seatunnel/plugin_config -v /mnt/sda1/seatunnel/apache-seatunnel-2.3.3/lib:/opt/seatunnel/lib -v /mnt/sda1/seatunnel/apache-seatunnel-2.3.3/plugins:/opt/seatunnel/plugins -v /mnt/sda1/seatunnel/apache-seatunnel-2.3.3/connectors/seatunnel:/opt/seatunnel/connectors/seatunnel -v /etc/localtime:/etc/localtime seatunnel:2.3.3

    #使用JDBC模式,同步两个库的数据

    • 修改plugin_config
        • --seatunnel-connectors--
        • connector-fake
        • connector-console
        • connector-jdbc
        • --end--
      • 安装插件,安装过程中会自动把相应的jar包复制到对应的目录中
        • sh bin/install-plugin.sh 2.3.3
      • 修改v2.streaming.conf.template配置

      Defining the runtime environment

      env {

      You can set flink configuration here

      execution.parallelism = 1
      job.mode = "BATCH"
      }
      source{
      Jdbc {
      url = "jdbc:mysql://mysql:3306/test"
      driver = "com.mysql.cj.jdbc.Driver"
      connection_check_timeout_sec = 100
      user = "root"
      password = "123456"
      query = "select * from help_keyword_1 limit 2"
      }
      }

      transform {
      # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
      # please go to https://seatunnel.apache.org/docs/transform-v2/sql
      }

      sink {

      Console {}

      jdbc {
      url = "jdbc:mysql://mysql:3306/test2"
      driver = "com.mysql.cj.jdbc.Driver"
      user = "root"
      password = "123456"
      query = "insert into help_keyword_1(help_keyword_1_id,name) values(?,?)"
      }
      }

#配置MYSQL-CDC通过binlog实时数据同步

    • 临时启动容器
      • docker run --name seatunnel --hostname seatunnel-node1 --network my-net -e config="/data/seatunnel.streaming.conf" -v /mnt/sda1/seatunnel/apache-seatunnel-2.3.3/config/v2.streaming.conf.template:/data/seatunnel.streaming.conf -v /mnt/sda1/seatunnel/apache-seatunnel-2.3.3/config/plugin_config:/opt/seatunnel/plugin_config -v /mnt/sda1/seatunnel/apache-seatunnel-2.3.3/lib:/opt/seatunnel/lib -v /mnt/sda1/seatunnel/apache-seatunnel-2.3.3/plugins:/opt/seatunnel/plugins -v /mnt/sda1/seatunnel/apache-seatunnel-2.3.3/connectors/seatunnel:/opt/seatunnel/connectors/seatunnel -v /etc/localtime:/etc/localtime seatunnel:2.3.3
相关推荐
likfishdn2 分钟前
Linux的文件与目录管理
linux·运维·服务器
chian-ocean22 分钟前
Linux 文件缓冲区:高效数据访问的幕后推手
linux·运维·服务器
加油=^_^=24 分钟前
【Linux】进程优先级 | 进程调度(三)
linux·运维·服务器
Dontla1 小时前
Docker run --add-host参数解析(在容器启动时向/etc/hosts文件中添加自定义的主机名与IP映射)(适用于临时调试或测试)
tcp/ip·docker·容器
西西小飞龙1 小时前
Docker和K8S中pod、services、container的介绍和关系
docker·容器·kubernetes
元气满满的热码式1 小时前
Docker数据卷操作实战
linux·运维·docker·云原生·容器
kngines1 小时前
【实战 ES】实战 Elasticsearch:快速上手与深度实践-1.3.1单节点安装(Docker与手动部署)
大数据·elasticsearch·docker
北京耐用通信1 小时前
航空装配自动化神器Ethercat转profient网关搭配机器人精准控制
运维·机器人·自动化
吃汤圆的抹香鲸1 小时前
Rider 安装包 绿色版 Win/Mac/Linux 适合.NET和游戏开发者使用 2025全栈开发终极指南:从零配置到企业级实战
linux·运维·windows·sql·游戏·macos·.net
崔lc4 小时前
linux中安装部署Jenkins,成功构建springboot项目详细教程
运维·jenkins