SeaTunnel Web安装 一把成

安装相关jar包,以及SeaTunnel 和Web 打成的包,可以直接使用,但是需要安装MySQL客户端的分享:

链接:https://pan.baidu.com/s/1qrt1RAX38SgIpNklbQJ7pA

提取码:0kmf

1. 环境准备

|----------|-----------|
| 环境名称 | 版本 |
| 系统环境 | CentOS7.9 |
| Java环境 | JDK1.8 |

1.1 系统环境

1.2 Java环境安装

下载jdk1.8安装包并拷贝到服务器

bash 复制代码
# 进入文件夹
cd /usr/local
#  将jdk拷贝到该位置
#  解压文件
tar -zxvf jdk-8u381-linux-x64.tar.gz

修改环境变量

bash 复制代码
vim /etc/profile

在环境变量中添加Java环境

bash 复制代码
export JAVA_HOME=/usr/local/jdk1.8.0_381/
export CLASSPATH=$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin

重新加载环境变量

bash 复制代码
source /etc/profile

验证Java环境

bash 复制代码
java -version

1.3 SeaTunnel单节点安装

下载安装包

下载地址:

Apache SeaTunnel

下载SeaTunnel安装包

下载SeaTunnel Web安装包

创建安装SeaTunnel 目录

bash 复制代码
# 创建用于安装SeaTunnel 后台引擎的目录
mkdir -p /opt/seatunnel/backend
# 创建用户安装SeaTunnel Web前台页面的目录
mkdir -p /opt/seatunnel/web

拷贝并安装SeaTunnel

bash 复制代码
# 进入到SeaTunnel 后台安装目录,将下载好的安装包拷贝到该目录
[root@localhost backend]# pwd
/opt/seatunnel/backend
[root@localhost backend]# ls
apache-seatunnel-2.3.3-bin.tar.gz
# 安装SeaTunnel 后台
tar -zxvf apache-seatunnel-2.3.3-bin.tar.gz

# 进入到SeaTunnel Web页面程序安装目录,将下载号的安装包拷贝到该目录
[root@localhost web]# pwd
/opt/seatunnel/web
[root@localhost web]# ls
apache-seatunnel-web-1.0.0-bin.tar.gz
# 安装SeaTunnel Web页面程序
tar -zxvf apache-seatunnel-web-1.0.0-bin.tar.gz

将SeaTunnel 后台配置到环境变量中

bash 复制代码
# 打开环境变量文件
vim /etc/profile

# 在环境变量文件最后添加如下配置
export SEATUNNEL_HOME=/opt/seatunnel/backend/apache-seatunnel-2.3.3
export PATH=$PATH:$SEATUNNEL_HOME/bin

# 重新加载环境变量文件
source /etc/profile

安装需要的连接器

bash 复制代码
# 进入到seatunnel 安装目录下
[root@localhost seatunnel]# cd apache-seatunnel-2.3.3
# 进入到插件配置文件中          
[root@localhost apache-seatunnel-2.3.3]# vim config/plugin_config
# 删除多余连接器,留下需要用到的连接器
--connectors-v2--
connector-cdc-mysql
connector-cdc-sqlserver
connector-doris
connector-elasticsearch
connector-jdbc
--end--
# 退出保存连接器配置文件
:qw
# 在线安装拦截器
[root@localhost apache-seatunnel-2.3.3]# sh bin/install-plugin.sh 
# 等待安装,安装完成后可查看已经安装的连接器
[root@localhost apache-seatunnel-2.3.3]# ls connectors/seatunnel/*
connectors/seatunnel/connector-cdc-mysql-2.3.3.jar
connectors/seatunnel/connector-cdc-sqlserver-2.3.3.jar
connectors/seatunnel/connector-console-2.3.3.jar
connectors/seatunnel/connector-doris-2.3.3.jar
connectors/seatunnel/connector-elasticsearch-2.3.3.jar
connectors/seatunnel/connector-fake-2.3.3.jar
connectors/seatunnel/connector-jdbc-2.3.3.jar

注意:

connector-jdbc 连接器可以连接多种使用JDBC进行连接的多种数据库,如下:

|----------------------------------------------------------------------------------------------------------------------------------------|
| MySQL、PostgreSQL、DM、Phoenix 、SQLServer、Oracle、sqlite、GBase8a、StarRocks、db2、saphana、Doris、teradata、Redshift、Snowflake、Vertica、OceanBase |

拷贝需要的数据库连接jar包

bash 复制代码
# 进入SeaTunnel 后台安装目录
cd /opt/seatunnel/backend/apache-seatunnel-2.3.3
# 创建连接器对应的目录,用于存放数据库连接驱动
[root@localhost apache-seatunnel-2.3.3]# mkdir -p plugins/jdbc/lib/
# 将下载的数据库连接包上传到该目录下
[root@localhost lib]# pwd
/opt/seatunnel/backend/apache-seatunnel-2.3.3/plugins/jdbc/lib
[root@localhost lib]# ls
mysql-connector-java-8.0.28.jar  ojdbc8.jar  postgresql-42.6.0.jar

数据库连接包下载地址:

|------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 数据库 | 下载地址 |
| mysql | https://mvnrepository.com/artifact/mysql/mysql-connector-java |
| oracle | https://mvnrepository.com/artifact/com.oracle.database.jdbc/ojdbc8 |
| postgresql | https://mvnrepository.com/artifact/org.postgresql/postgresql |
| doris | https://mvnrepository.com/artifact/mysql/mysql-connector-java |

测试验证

bash 复制代码
# 进入SeaTunnel 后台安装目录
cd /opt/seatunnel/backend/apache-seatunnel-2.3.3
# 跑样例脚本
sh bin/seatunnel.sh --config config/v2.batch.config.template -e local

出现以下信息表示验证成功

后台启用服务

bash 复制代码
# 进入SeaTunnel 后台安装目录
cd /opt/seatunnel/backend/apache-seatunnel-2.3.3
# 启动服务
nohup sh bin/seatunnel-cluster.sh 2>&1 &

查看SeaTunnel 运行日志

bash 复制代码
# 查看SeaTunnel运行日志
tail -100f logs/seatunnel-engine-server.log 

日志中有以下信息表示后台启动成功

1.4 安装SeaTunnel Web

注意:必须保证SeaTunnel的后台引擎服务正常运行,Web端才可以正常运行。

安装MySQL客户端

因为SeaTunnel Web 需要连接到MySQL数据库,并初始化数据库,所以需要安装MySQL Client。

bash 复制代码
# 先检查服务器中是否存在mariadb
rpm -qa|grep mariadb
#删除相关依赖包(若不存在,可以跳过)
rpm -e --nodeps mariadb-libs
#检查是否删除干净
rpm -qa|grep mariadb
# 检查MySQL
rpm -qa|grep mysql

# 创建一个目录用于存放MySQL Client 安装文件
mkdir -p /opt/mysqlrpm
# 将安装包拷贝到该目录下
[root@localhost mysqlrpm]# pwd
/opt/mysqlrpm
[root@localhost mysqlrpm]# ls
lrzsz-0.12.20-36.el7.x86_64.rpm
mysql-community-client-5.7.40-1.el7.x86_64.rpm
mysql-community-common-5.7.40-1.el7.x86_64.rpm
mysql-community-libs-5.7.40-1.el7.x86_64.rpm
# 安装lrzsz 
rpm -ivh lrzsz-0.12.20-36.el7.x86_64.rpm
# 按照顺序安装MySQL Client 
rpm -ivh mysql-community-common-5.7.40-1.el7.x86_64.rpm 
rpm -ivh mysql-community-libs-5.7.40-1.el7.x86_64.rpm
rpm -ivh mysql-community-client-5.7.40-1.el7.x86_64.rpm 

修改SeaTunnel Web 下script/seatunnel_server_env.sh中连接数据库信息

bash 复制代码
# 进入到SeaTunnel Web 安装目录
cd /opt/seatunnel/web/apache-seatunnel-web-1.0.0-bin
# 备份script/seatunnel_server_env.sh文件
cp script/seatunnel_server_env.sh script/seatunnel_server_env.sh.bak
# 修改script/seatunnel_server_env.sh文件
vim script/seatunnel_server_env.sh
# 修改以下配置信息
export HOSTNAME="127.0.0.1"
export PORT="3306"
export USERNAME="root"
export PASSWORD="123456"
# 修改如下,加上了前缀,以防与系统中其它配置冲突
# 将数据库连接信息修改成你要链接的信息
export STWEB_HOSTNAME="127.0.0.1"
export STWEB_PORT="3306"
export STWEB_USERNAME="root"
export STWEB_PASSWORD="123456"

修改SeaTunnel Web 下script/init_sql.sh 中连接数据库信息

bash 复制代码
vim script/init_sql.sh

主要是保证里边的${XXXXX}的配置信息与script/seatunnel_server_env.sh中的保持一致。

执行数据库初始化脚本

bash 复制代码
# 进入SeaTunnel Web 安装目录
cd /opt/seatunnel/web/apache-seatunnel-web-1.0.0-bin
# 执行初始化文件
sh script/init_sql.sh

修改SeaTunnel Web 连接MySQL数据库的配置信息

bash 复制代码
# 进入SeaTunnel Web 安装目录
cd /opt/seatunnel/web/apache-seatunnel-web-1.0.0-bin
# 打开SeaTunnel Web 配置文件
vim conf/application.yml
# 修改以下部分配置:端口号,MySQL连接IP,MySQL用户名和密码
server:
  #port: 8801
  port: 8882

spring:
  application:
    name: seatunnel
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
  datasource:
    driver-class-name: com.mysql.jdbc.Driver
    url: jdbc:mysql://192.168.1.111:3306/seatunnel?useSSL=false&useUnicode=true&characterEnc
oding=utf-8&allowMultiQueries=true&allowPublicKeyRetrieval=true    
    username: root
    password: 123456

配置client信息

bash 复制代码
# 将seatunnel后台的两个文件拷贝到web的conf目录下
cp /opt/seatunnel/backend/apache-seatunnel-2.3.3/config/hazelcast-client.yaml conf/
cp /opt/seatunnel/backend/apache-seatunnel-2.3.3/connectors/plugin-mapping.properties conf/

下载Web程序需要的数据源(数据源需要的各种jar包)

这里在各种jar包,如果缺失的话,在Web页面创建数据源时会出现空数据源的情况。

下载jar包

首先你自己的电脑上安装了github

下载jar包额度脚本

我是在我电脑的D盘根目录下,创建了download_datasource.sh文件,文件内容如下:

有两处需要修改,请看注释。

bash 复制代码
#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

#This script is used to download the connector plug-ins required during the running process.
#All are downloaded by default. You can also choose what you need.
#You only need to configure the plug-in name in config/plugin_config.

# get seatunnel web home
# SEATUNNEL_WEB_HOME=$(cd $(dirname $0);cd ../;pwd)
# 此处需要修改成你本地的目录
DATASOURCE_DIR=D:/down_jars

# If you don't want to download a certain data source, you can delete the element below
datasource_list=(
  "datasource-plugins-api"
  "datasource-elasticsearch"
  "datasource-hive"
  "datasource-jdbc-clickhouse"
  "datasource-jdbc-hive"
  "datasource-jdbc-mysql"
  "datasource-jdbc-oracle"
  "datasource-jdbc-postgresql"
  "datasource-jdbc-redshift"
  "datasource-jdbc-sqlserver"
  "datasource-jdbc-starrocks"
  "datasource-jdbc-tidb"
  "datasource-kafka"
  "datasource-mysql-cdc"
  "datasource-s3"
  "datasource-sqlserver-cdc"
  "datasource-starrocks"
)

# the datasource default version is 1.0.0, you can also choose a custom version. eg: 1.1.2:  sh install-datasource.sh 2.1.2
version=1.0.0

if [ -n "$1" ]; then
    version="$1"
fi

echo "Downloading SeaTunnel Web Datasource lib, usage version is ${version}"

# create the datasource directory
if [ ! -d "$DATASOURCE_DIR" ];
  then
      mkdir -p "$DATASOURCE_DIR"
      echo "Created datasource directory."
fi

for i in "${datasource_list[@]}"
do
	echo "$i"
	echo "Downloading datasource: " "$i"
# 修改成mvn
  mvn dependency:get -DgroupId=org.apache.seatunnel -DartifactId="$i" -Dversion="$version" -Ddest="$DATASOURCE_DIR"
done

之后在download_datasource.sh目录下点击鼠标右键,选择Git Bash Here

之后在打开的页面执行 sh download_datasource.sh 点击回车,等待下载完成。

之后将下载下来的包,拷贝到SeaTunnel Web 程序的 /libs目录下。

将MySQL Jar 包拷贝到libs目录下

将mysql连接数据库的Jar 拷贝到**/libs** 目录下,否则Web程序启动时会报错,连不上数据库。

启动SeaTunnel Web 程序

bash 复制代码
# 进入SeaTunnel Web 安装目录
cd /opt/seatunnel/web/apache-seatunnel-web-1.0.0-bin
# 启动SeaTunnel Web
sh bin/seatunnel-backend-daemon.sh start
# 查看启动日志
tail -100f logs/seatunnel-web.log

配置开放端口号(两种方式)

bash 复制代码
#-------------iptables-----------------
#开打端口号开放配置文件
vim /etc/sysconfig/iptables
# 添加一行
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8882 -j ACCEPT
# 重启启动防火墙
systemctl restart iptables.service
#------------firewall-------------------
# 依次执行下面命令
firewall-cmd --zone=public --add-port=8882/tcp --permanent
firewall-cmd --reload
firewall-cmd --list-all

查看日志文件

访问SeaTunnel Web

http://192.168.182.166:8882

默认用户名密码:admin/admin

到此为止SeaTunnel 与SeaTunnel Web 已经安装完成。

相关推荐
ApacheSeaTunnel5 天前
保姆级 SeaTunnel 入门!再学不会小编当场表演倒立敲代码
大数据·数据集成·seatunnel·技术分享·数据同步
SeaTunnel10 天前
Apache SeaTunnel MySQL CDC 支持按时间启动吗?
大数据·数据库·mysql·开源·apache·seatunnel
ApacheSeaTunnel12 天前
告别手敲 Schema!SeaTunnel 集成 Gravitino 元数据 RestApi 这个新动作有点酷
大数据·ai·seatunnel·技术分享·数据同步·gravitino
SeaTunnel24 天前
Apache SeaTunnel 2025 案例精选重磅发布!
大数据·开源·apache·seatunnel·案例
码农丁丁2 个月前
apache seatunnel 2.3.12 Changelog
apache·seatunnel
SeaTunnel2 个月前
Apache SeaTunnel 如何将 CDC 数据流转换为 Append-Only 模式?
大数据·开源·apache·开发者·seatunnel·转换插件
SeaTunnel3 个月前
结项报告完整版:Apache SeaTunnel 支持 Flink 引擎 Schema Evolution 功能
java·大数据·flink·开源·seatunnel
ApacheSeaTunnel3 个月前
结项报告完整版 | Apache SeaTunnel支持metalake开发
大数据·开源·数据集成·seatunnel·数据同步
SeaTunnel3 个月前
Apache SeaTunnel 支持 Metalake 开发了!避免任务配置敏感信息暴露
大数据·开源·apache·个人开发·数据集成·seatunnel·看开源之夏
ApacheSeaTunnel4 个月前
从小时级到分钟级:多点DMALL如何用Apache SeaTunnel把数据集成成本砍到1/3?
大数据·开源·数据集成·seatunnel·技术分享