Apache DolphinScheduler 如何实现自动化打包+单机/集群部署？

DolphinScheduler社区2023-09-19 8:53

Apache DolphinScheduler 是一款开源的分布式任务调度系统，旨在帮助用户实现复杂任务的自动化调度和管理。DolphinScheduler 支持多种任务类型，可以在单机或集群环境下运行。下面将介绍如何实现 DolphinScheduler 的自动化打包和单机/集群部署。

自动化打包

所需环境：maven、jdk

执行以下shell完成代码拉取及打包，打包路径：/opt/action/dolphinscheduler/dolphinscheduler-dist/target/apache-dolphinscheduler-dev-SNAPSHOT-bin.tar.gz

clone_init(){​ 复制代码 
sudo su - root <<EOF
cd /opt/action
git clone git@github.com:apache/dolphinscheduler.git
cd Dolphinscheduler
git fetch origin dev
git checkout -b dev origin/dev
#git log --oneline
EOF
}

# 打包
build(){
sudo su - root <<EOF
cd /opt/action/Dolphinscheduler
mvn -B clean install -Prelease -Dmaven.test.skip=true -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true
EOF
}

单机部署

1、DolphinScheduler运行所需环境

所需环境jdk、zookeeper、mysql

初始化zookeeper（高版本zookeeper推荐使用v3.8及以上版本）环境

安装包官网下载地址：https://zookeeper.apache.org/

复制代码 
sudo su - root <<EOF
#进入/opt目录下（安装目录自行选择）
cd /opt
#解压缩
tar -xvf apache-zookeeper-3.8.0-bin.tar.gz
#修改文件名称
sudo mv apache-zookeeper-3.8.0-bin zookeeper
#进入zookeeper目录
cd zookeeper/
#在 /opt/zookeeper 目录下创建目录 zkData，用来存放 zookeeper 的数据文件
mkdir zkData
#进入conf文件夹
cd conf/
#修改配置文件，复制 zoo_sample.cfg 文件并重命名为 zoo.cfg因为zookeeper只能识别 zoo.cfg 配置文件
cp zoo_sample.cfg zoo.cfg
#修改 zoo.cfg 的配置
sed  -i 's/\/tmp\/zookeeper/\/opt\/zookeeper\/conf/g' zoo.cfg
#停止之前的zk服务
ps -ef|grep QuorumPeerMain|grep -v grep|awk '{print "kill -9 " $2}' |sh
#使用 vim zoo.cfg 命令修改 zoo.cfg 的配置
sh /opt/zookeeper/bin/zkServer.sh start
EOF
}

jdk、mysql这里不做过多赘述。

2、初始化配置

2.1 配置文件初始化

初始化文件要放到指定目录（本文章以/opt/action/tool举例）

  • 2.1.1新建文件夹

    复制代码 
    mkdir -p /opt/action/tool
mkdir -p /opt/Dsrelease

  • 2.1.2新建初始化文件common.properties

    Licensed to the Apache Software Foundation (ASF) under one or more

    contributor license agreements. See the NOTICE file distributed with

    this work for additional information regarding copyright ownership.

    The ASF licenses this file to You under the Apache License, Version 2.0

    (the "License"); you may not use this file except in compliance with

    the License. You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software

    distributed under the License is distributed on an "AS IS" BASIS,

    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

    See the License for the specific language governing permissions and

    limitations under the License.

    user data local directory path, please make sure the directory exists and have read write permissions

    data.basedir.path=/tmp/dolphinscheduler

    resource storage type: HDFS, S3, NONE

    resource.storage.type=HDFS

    resource store on HDFS/S3 path, resource file will store to this hadoop hdfs path, self configuration, please make sure the directory exists on hdfs and have read write permissions. "/dolphinscheduler" is recommended

    resource.upload.path=/dolphinscheduler

    whether to startup kerberos

    hadoop.security.authentication.startup.state=false

    java.security.krb5.conf path

    java.security.krb5.conf.path=/opt/krb5.conf

    login user from keytab username

    login.user.keytab.username=hdfs-mycluster@ESZ.COM

    login user from keytab path

    login.user.keytab.path=/opt/hdfs.headless.keytab

    kerberos expire time, the unit is hour

    kerberos.expire.time=2

    resource view suffixs

    #resource.view.suffixs=txt,log,sh,bat,conf,cfg,py,java,sql,xml,hql,properties,json,yml,yaml,ini,js

    if resource.storage.type=HDFS, the user must have the permission to create directories under the HDFS root path

    hdfs.root.user=root

    if resource.storage.type=S3, the value like: s3a://dolphinscheduler; if resource.storage.type=HDFS and namenode HA is enabled, you need to copy core-site.xml and hdfs-site.xml to conf dir

    fs.defaultFS=file:///
    aws.access.key.id=minioadmin
    aws.secret.access.key=minioadmin
    aws.region=us-east-1
    aws.endpoint=http://localhost:9000

    resourcemanager port, the default value is 8088 if not specified

    resource.manager.httpaddress.port=8088

    if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single, keep this value empty

    yarn.resourcemanager.ha.rm.ids=192.168.xx.xx,192.168.xx.xx

    if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single, you only need to replace aws2 to actual resourcemanager hostname

    yarn.application.status.address=http://aws2:%s/ws/v1/cluster/apps/%s

    job history status url when application number threshold is reached(default 10000, maybe it was set to 1000)

    yarn.job.history.status.address=http://aws2:19888/ws/v1/history/mapreduce/jobs/%s

    datasource encryption enable

    datasource.encryption.enable=false

    datasource encryption salt

    datasource.encryption.salt=!@#$%^&*

    data quality option

    data-quality.jar.name=dolphinscheduler-data-quality-dev-SNAPSHOT.jar

    #data-quality.error.output.path=/tmp/data-quality-error-data

    Network IP gets priority, default inner outer

    Whether hive SQL is executed in the same session

    support.hive.oneSession=false

    use sudo or not, if set true, executing user is tenant user and deploy user needs sudo permissions; if set false, executing user is the deploy user and doesn't need sudo permissions

    sudo.enable=true

    network interface preferred like eth0, default: empty

    #dolphin.scheduler.network.interface.preferred=

    network IP gets priority, default: inner outer

    #dolphin.scheduler.network.priority.strategy=default

    system env path

    #dolphinscheduler.env.path=dolphinscheduler_env.sh

    development state

    development.state=false

    rpc port

    alert.rpc.port=50052

    Url endpoint for zeppelin RESTful API

    zeppelin.rest.url=http://localhost:8080

  • 2.1.3新建初始化文件core-site.xml

    fs.defaultFS hdfs://aws1 ha.zookeeper.quorum aws1:2181 hadoop.proxyuser.root.hosts * hadoop.proxyuser.root.groups *

  • 2.1.4新建初始化文件hdfs-site.xml

    dfs.replication 1 dfs.namenode.name.dir /opt/bigdata/hadoop/ha/dfs/name dfs.datanode.data.dir /opt/bigdata/hadoop/ha/dfs/data dfs.namenode.secondary.http-address aws2:50090 dfs.namenode.checkpoint.dir /opt/bigdata/hadoop/ha/dfs/secondary dfs.nameservices aws1 dfs.ha.namenodes.aws1 nn1,nn2 dfs.namenode.rpc-address.aws1.nn1 aws1:8020 dfs.namenode.rpc-address.aws1.nn2 aws2:8020 dfs.namenode.http-address.aws1.nn1 aws1:50070 dfs.namenode.http-address.aws1.nn2 aws2:50070 dfs.datanode.address aws1:50010 dfs.datanode.ipc.address aws1:50020 dfs.datanode.http.address aws1:50075 dfs.datanode.https.address aws1:50475

    dfs.namenode.shared.edits.dir
    qjournal://aws1:8485;aws2:8485;aws3:8485/mycluster


    dfs.journalnode.edits.dir
    /opt/bigdata/hadoop/ha/dfs/jn


    dfs.client.failover.proxy.provider.aws1
    org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider


    dfs.ha.fencing.methods
    sshfence


    dfs.ha.fencing.ssh.private-key-files
    /root/.ssh/id_dsa


    dfs.ha.automatic-failover.enabled
    true

  • 2.1.5上传初始化jar包mysql-connector-java-8.0.16.jar

  • 2.1.6上传初始化jar包ojdbc8.jar

2.2 初始化文件替换

复制代码 
cd /opt/Dsrelease
sudo rm -r $today/

echo "rm -r $today"

cd /opt/release

cp $packge_tar /opt/Dsrelease

cd /opt/Dsrelease

tar -zxvf $packge_tar
mv $packge  $today
p_api_lib=/opt/Dsrelease/$today/api-server/libs/
p_master_lib=/opt/Dsrelease/$today/master-server/libs/
p_worker_lib=/opt/Dsrelease/$today/worker-server/libs/
p_alert_lib=/opt/Dsrelease/$today/alert-server/libs/
p_tools_lib=/opt/Dsrelease/$today/tools/libs/
p_st_lib=/opt/Dsrelease/$today/standalone-server/libs/


p_api_conf=/opt/Dsrelease/$today/api-server/conf/
p_master_conf=/opt/Dsrelease/$today/master-server/conf/
p_worker_conf=/opt/Dsrelease/$today/worker-server/conf/
p_alert_conf=/opt/Dsrelease/$today/alert-server/conf/
p_tools_conf=/opt/Dsrelease/$today/tools/conf/
p_st_conf=/opt/Dsrelease/$today/standalone-server/conf/


cp $p0 $p4 $p_api_lib
cp $p0 $p4 $p_master_lib
cp $p0 $p4 $p_worker_lib
cp $p0 $p4 $p_alert_lib
cp $p0 $p4 $p_tools_lib
cp $p0 $p4 $p_st_lib

echo "cp $p0 $p_api_lib"

cp $p1 $p2 $p3 $p_api_conf
cp $p1 $p2 $p3 $p_master_conf
cp $p1 $p2 $p3 $p_worker_conf
cp $p1 $p2 $p3 $p_alert_conf
cp $p1 $p2 $p3 $p_tools_conf
cp $p1 $p2 $p3 $p_st_conf



echo "cp $p1 $p2 $p3 $p_api_conf"
}

define_param(){

packge_tar=apache-dolphinscheduler-dev-SNAPSHOT-bin.tar.gz
packge=apache-dolphinscheduler-dev-SNAPSHOT-bin
p0=/opt/action/tool/mysql-connector-java-8.0.16.jar
p1=/opt/action/tool/common.properties
p2=/opt/action/tool/core-site.xml
p3=/opt/action/tool/hdfs-site.xml
p4=/opt/action/tool/ojdbc8.jar

today=`date +%m%d`


}

2.3 配置文件内容替换

复制代码 
sed  -i 's/spark2/spark/g' /opt/Dsrelease/$today/worker-server/conf/dolphinscheduler_env.sh

cd /opt/Dsrelease/$today/bin/env/
sed -i '$a\export SPRING_PROFILES_ACTIVE=permission_shiro' dolphinscheduler_env.sh
sed -i '$a\export DATABASE="mysql"' dolphinscheduler_env.sh
sed -i '$a\export SPRING_DATASOURCE_DRIVER_CLASS_NAME="com.mysql.jdbc.Driver"' dolphinscheduler_env.sh
#自定义修改mysql配置
sed -i '$a\export SPRING_DATASOURCE_URL="jdbc:mysql://ctyun6:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&allowMultiQueries=true&allowPublicKeyRetrieval=true"' dolphinscheduler_env.sh
sed -i '$a\export SPRING_DATASOURCE_USERNAME="root"' dolphinscheduler_env.sh
sed -i '$a\export SPRING_DATASOURCE_PASSWORD="root@123"' dolphinscheduler_env.sh
echo "替换jdbc配置成功"
#自定义修改zookeeper配置
sed -i '$a\export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper}' dolphinscheduler_env.sh
sed -i '$a\export REGISTRY_ZOOKEEPER_CONNECT_STRING=${REGISTRY_ZOOKEEPER_CONNECT_STRING:-ctyun6:2181}' dolphinscheduler_env.sh
echo "替换zookeeper配置成功"
sed -i 's/resource.storage.type=HDFS/resource.storage.type=NONE/' /opt/Dsrelease/$today/master-server/conf/common.properties
sed -i 's/resource.storage.type=HDFS/resource.storage.type=NONE/' /opt/Dsrelease/$today/worker-server/conf/common.properties
sed -i 's/resource.storage.type=HDFS/resource.storage.type=NONE/' /opt/Dsrelease/$today/alert-server/conf/common.properties
sed -i 's/resource.storage.type=HDFS/resource.storage.type=NONE/' /opt/Dsrelease/$today/api-server/conf/common.properties
sed -i 's/hdfs.root.user=root/resource.hdfs.root.user=root/' /opt/Dsrelease/$today/master-server/conf/common.properties
sed -i 's/hdfs.root.user=root/resource.hdfs.root.user=root/' /opt/Dsrelease/$today/worker-server/conf/common.properties
sed -i 's/hdfs.root.user=root/resource.hdfs.root.user=root/' /opt/Dsrelease/$today/alert-server/conf/common.properties
sed -i 's/hdfs.root.user=root/resource.hdfs.root.user=root/' /opt/Dsrelease/$today/api-server/conf/common.properties
sed -i 's/fs.defaultFS=file:/resource.fs.defaultFS=file:/' /opt/Dsrelease/$today/master-server/conf/common.properties
sed -i 's/fs.defaultFS=file:/resource.fs.defaultFS=file:/' /opt/Dsrelease/$today/worker-server/conf/common.properties
sed -i 's/fs.defaultFS=file:/resource.fs.defaultFS=file:/' /opt/Dsrelease/$today/alert-server/conf/common.properties
sed -i 's/fs.defaultFS=file:/resource.fs.defaultFS=file:/' /opt/Dsrelease/$today/api-server/conf/common.properties
sed -i '$a\resource.hdfs.fs.defaultFS=file:///' /opt/Dsrelease/$today/api-server/conf/common.properties
echo "替换common.properties配置成功"
# 替换master worker内存 api alert也可进行修改，具体根据当前服务器硬件配置而定，但要遵循Xms=Xmx=2Xmn的规律
cd /opt/Dsrelease/$today/
sed -i 's/Xms4g/Xms2g/g' worker-server/bin/start.sh
sed -i 's/Xmx4g/Xmx2g/g' worker-server/bin/start.sh
sed -i 's/Xmn2g/Xmn1g/g' worker-server/bin/start.sh
sed -i 's/Xms4g/Xms2g/g' master-server/bin/start.sh
sed -i 's/Xmx4g/Xmx2g/g' master-server/bin/start.sh
sed -i 's/Xmn2g/Xmn1g/g' master-server/bin/start.sh
echo "master worker内存修改完成"

}

3、删除HDFS配置

复制代码 
echo "开始删除hdfs配置"
sudo rm /opt/Dsrelease/$today/api-server/conf/core-site.xml
sudo rm /opt/Dsrelease/$today/api-server/conf/hdfs-site.xml
sudo rm /opt/Dsrelease/$today/worker-server/conf/core-site.xml
sudo rm /opt/Dsrelease/$today/worker-server/conf/hdfs-site.xml
sudo rm /opt/Dsrelease/$today/master-server/conf/core-site.xml
sudo rm /opt/Dsrelease/$today/master-server/conf/hdfs-site.xml
sudo rm /opt/Dsrelease/$today/alert-server/conf/core-site.xml
sudo rm /opt/Dsrelease/$today/alert-server/conf/hdfs-site.xml
echo "结束删除hdfs配置"
}

4、MySQL初始化

复制代码 
init_mysql(){

sql_path="/opt/Dsrelease/$today/tools/sql/sql/dolphinscheduler_mysql.sql"
sourceCommand="source $sql_path"
echo $sourceCommand
echo "开始source："
mysql -hlocalhost -uroot -proot@123 -D "dolphinscheduler" -e "$sourceCommand"
echo "结束source："
}

5、启动DolphinScheduler服务

复制代码 
stop_all_server(){
cd /opt/Dsrelease/$today
./bin/dolphinscheduler-daemon.sh stop api-server
./bin/dolphinscheduler-daemon.sh stop master-server
./bin/dolphinscheduler-daemon.sh stop worker-server
./bin/dolphinscheduler-daemon.sh stop alert-server
ps -ef|grep api-server|grep -v grep|awk '{print "kill -9 " $2}' |sh
ps -ef|grep master-server |grep -v grep|awk '{print "kill -9 " $2}' |sh
ps -ef|grep worker-server |grep -v grep|awk '{print "kill -9 " $2}' |sh
ps -ef|grep alert-server |grep -v grep|awk '{print "kill -9 " $2}' |sh
}

run_all_server(){
cd /opt/Dsrelease/$today
./bin/dolphinscheduler-daemon.sh start api-server
./bin/dolphinscheduler-daemon.sh start master-server
./bin/dolphinscheduler-daemon.sh start worker-server
./bin/dolphinscheduler-daemon.sh start alert-server
}

集群部署

1、开放mysql和zookeeper对外端口 2、集群部署及启动

复制完成初始化的文件夹到指定的服务器，启动指定服务即可完成集群部署，要连同一个Zookeeper和MySQL。

本文由 白鲸开源科技 提供发布支持！

相关推荐
大模型码小白10 小时前
【Python零基础教程】继承、多态与魔法函数：面向对象编程三大核心特性详解
java·大数据·开发语言·人工智能·python·ai编程
延凡科技13 小时前
多场景落地复盘：端边云架构无人机智能巡检系统设计与实践
大数据·数据结构·人工智能·科技·架构·无人机·能源
ifenxi爱分析14 小时前
爱分析最新报告解读：AI数据基础设施与数据中台的区别
大数据·人工智能
品牌全球行15 小时前
共商共建共享 链接数字未来——“一带一路数字新城（深圳）会客厅筹备办”揭牌仪式在深圳隆重举行
大数据·人工智能
数智化管理手记16 小时前
账龄分析手工统计易遗漏？自动账龄分析工具怎么搭建
大数据·网络·数据库·人工智能·数据挖掘
ApacheSeaTunnel17 小时前
Apache SeaTunnel AI CLI Benchmark：7 款大模型、100 个 ETL 任务实测，谁真正能跑起来？
大数据·ai·开源·大模型·数据集成·cli·seatunnel·技术分享·数据同步
电商API_1800790524717 小时前
企业ERP进销存场景｜京东商品详情接口自动同步方案｜凭证鉴权批量调用技术实操
大数据·运维·人工智能·爬虫·数据挖掘·网络爬虫
小稻穗18 小时前
市场调研样本选型坐标系：2026六家国内样本平台能力横向校验
大数据·运维·数据分析
极光代码工作室18 小时前
基于Spark的日志监控与分析平台
大数据·hadoop·python·spark·数据可视化
wuhanzhanhui19 小时前
从高强钢到碳纤维！2026武汉汽车材料轻量化制造技术展会，重塑未来造车新范式
大数据·汽车·制造
热门推荐
012026年7月AI圈大地震：GPT-5.6被政府限制、Claude入驻Slack、Anthropic自研芯片02GitHub 镜像站点03如何新建文件夹？ 电脑新建文件夹的4种方法04AI科技热点日报 | 2026年07月01日05幻兽帕鲁 - 服务器管理员权限与 GM 命令完全指南06国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）072026 国产 AI 大模型横评：DeepSeek、通义千问、Kimi、文心一言、星火、豆包谁更能打？08AI 编程 IDE 全景解析 2026：Agent 全面接管开发链路092026 年 AI 大模型 & AI 编程工具实战全总结10微信历史版本含下载地址（ Windows PC | 安卓 | MAC ）及设置微信不更新