阿里云CentOS7安装Flink1.17

前提条件

阿里云CentOS7安装好jdk,官方文档要求java 11,使用java 8也可以。可参 hadoop安装 的jdk安装部分。

下载安装包

下载安装包

复制代码
[hadoop@node1 ~]$ cd softinstall/
[hadoop@node1 softinstall]$ wget https://archive.apache.org/dist/flink/flink-1.17.1/flink-1.17.1-bin-scala_2.12.tgz

查看安装包

复制代码
[hadoop@node1 softinstall]$ ls
apache-zookeeper-3.7.1-bin.tar.gz  hadoop-3.3.4.tar.gz         kafka_2.12-3.3.1.tgz
flink-1.17.1-bin-scala_2.12.tgz    jdk-8u271-linux-x64.tar.gz

解压安装包

解压安装包

复制代码
[hadoop@node1 softinstall]$ tar -zxvf flink-1.17.1-bin-scala_2.12.tgz -C ~/soft

查看解压后的文件

复制代码
[hadoop@node1 softinstall]$ cd ~/soft
[hadoop@node1 soft]$ ls
apache-zookeeper-3.7.1-bin  flink-1.17.1  hadoop-3.3.4  jdk1.8.0_271  kafka_2.12-3.3.1

设置环境变量

复制代码
[hadoop@node1 soft]$ sudo nano /etc/profile.d/my_env.sh

添加如下内容

复制代码
export FLINK_HOME=/home/hadoop/soft/flink-1.17.1
export PATH=$PATH:$FLINK_HOME/bin

让环境变量立即生效

复制代码
[hadoop@node1 soft]$ source /etc/profile

验证版本号

复制代码
[hadoop@node1 soft]$ flink -v
Version: 1.17.1, Commit ID: 2750d5c

查看Flink配置文件

复制代码
[hadoop@node1 soft]$ cd $FLINK_HOME
​
[hadoop@node1 flink]$ ls
bin  conf  examples  lib  LICENSE  licenses  log  NOTICE  opt  plugins  README.txt
​
[hadoop@node1 flink]$ cd conf
[hadoop@node1 conf]$ ls
flink-conf.yaml       log4j-console.properties  log4j-session.properties  logback-session.xml  masters  zoo.cfg
log4j-cli.properties  log4j.properties          logback-console.xml       logback.xml          workers
​
复制代码
[hadoop@node1 conf]$ vim flink-conf.yaml

找到如下配置项,修改内容如下,其中node1是机器主机名,注意根据实际情况修改。

复制代码
jobmanager.rpc.address: node1
jobmanager.bind-host: 0.0.0.0
taskmanager.bind-host: 0.0.0.0
taskmanager.host: node1
rest.address: node1
rest.bind-address: 0.0.0.0

配置master

复制代码
[hadoop@node1 conf]$ nano masters

修改内容为

复制代码
node1:8081

配置worker

复制代码
[hadoop@node1 conf]$ nano workers

修改内容为

复制代码
node1

启动/关闭flink集群

查看flink的命令

复制代码
[hadoop@node1 conf]$ cd $FLINK_HOME/bin
[hadoop@node1 bin]$ ls
bash-java-utils.jar  flink-daemon.sh           kubernetes-taskmanager.sh  start-cluster.sh           yarn-session.sh
config.sh            historyserver.sh          pyflink-shell.sh           start-zookeeper-quorum.sh  zookeeper.sh
find-flink-home.sh   jobmanager.sh             sql-client.sh              stop-cluster.sh
flink                kubernetes-jobmanager.sh  sql-gateway.sh             stop-zookeeper-quorum.sh
flink-console.sh     kubernetes-session.sh     standalone-job.sh          taskmanager.sh
[hadoop@node1 bin]$ 
​

启动flink集群

复制代码
[hadoop@node1 flink-1.17.1]$ start-cluster.sh 
Starting cluster.
Starting standalonesession daemon on host node1.
Starting taskexecutor daemon on host node1.

查看进程

复制代码
[hadoop@node1 flink-1.17.1]$ jps
13527 Jps
13177 StandaloneSessionClusterEntrypoint
13503 TaskManagerRunner

关闭flink集群

复制代码
[hadoop@node1 flink-1.17.1]$ stop-cluster.sh 
Stopping taskexecutor daemon (pid: 13503) on host node1.
Stopping standalonesession daemon (pid: 13177) on host node1.

查看进程

复制代码
[hadoop@node1 flink-1.17.1]$ jps
14760 Jps

单独启动flink进程

复制代码
[hadoop@node1 flink-1.17.1]$ jobmanager.sh start
[hadoop@node1 flink-1.17.1]$ taskmanager.sh start

操作过程如下

复制代码
[hadoop@node1 flink-1.17.1]$ jobmanager.sh 
Usage: jobmanager.sh ((start|start-foreground) [args])|stop|stop-all
[hadoop@node1 flink-1.17.1]$ jobmanager.sh start
Starting standalonesession daemon on host node1.
[hadoop@node1 flink-1.17.1]$ jps
15300 StandaloneSessionClusterEntrypoint
15316 Jps
[hadoop@node1 flink-1.17.1]$ taskmanager.sh 
Usage: taskmanager.sh (start|start-foreground|stop|stop-all)
[hadoop@node1 flink-1.17.1]$ taskmanager.sh start
Starting taskexecutor daemon on host node1.
[hadoop@node1 flink-1.17.1]$ jps
15300 StandaloneSessionClusterEntrypoint
15668 TaskManagerRunner
15693 Jps

单独关闭flink进程

复制代码
[hadoop@node1 flink-1.17.1]$ taskmanager.sh stop
[hadoop@node1 flink-1.17.1]$ jobmanager.sh stop

操作过程如下

复制代码
[hadoop@node1 flink-1.17.1]$ taskmanager.sh stop
Stopping taskexecutor daemon (pid: 15668) on host node1.
[hadoop@node1 flink-1.17.1]$ jps
15300 StandaloneSessionClusterEntrypoint
16232 Jps
[hadoop@node1 flink-1.17.1]$ jobmanager.sh stop
Stopping standalonesession daemon (pid: 15300) on host node1.
[hadoop@node1 flink-1.17.1]$ jps
16544 Jps

访问Web UI

启动flink集群

复制代码
start-cluster.sh

在阿里云的安全组放开8081端口

访问

复制代码
云服务器公网IP:8081

提交flink作业

运行flink提供的WordCount程序案例

复制代码
[hadoop@node1 examples]$ cd $FLINK_HOME
[hadoop@node1 flink-1.17.1]$ flink run examples/streaming/WordCount.jar
Executing example with default input data.
Use --input to specify file input.
Printing result to stdout. Use --output to specify output path.
Job has been submitted with JobID c4fcf3a5266af2d7e0df713d5bfdce85
Program execution finished
Job with JobID c4fcf3a5266af2d7e0df713d5bfdce85 has finished.
Job Runtime: 3139 ms

查看作业结果

查看输出的WordCount结果的末尾10行数据

复制代码
[hadoop@node1 flink-1.17.1]$ tail log/flink-*-taskexecutor-*.out
==> log/flink-hadoop-taskexecutor-0-node1.out <==
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000e0000000, 536870912, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 536870912 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home/hadoop/hs_err_pid10313.log
​
==> log/flink-hadoop-taskexecutor-1-node1.out <==
(nymph,1)
(in,3)
(thy,1)
(orisons,1)
(be,4)
(all,2)
(my,1)
(sins,1)
(remember,1)
(d,4)

Web UI查看提交的作业

在Task Managers查看作业输出结果

遇到的问题

启动flink,jps进程不对,查看log

复制代码
[hadoop@node1 log]$ cat flink-hadoop-standalonesession-0-node1.out 
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000d5550000, 715849728, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 715849728 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home/hadoop/soft/flink-1.17.1/bin/hs_err_pid10001.log
​

云服务器内存不足,关闭一些开启的进程来释放资源,或者提升云服务器配置,即可解决。

参考:First steps | Apache Flink

完成!enjoy it!

相关推荐
阿里云大数据AI技术2 小时前
Flink Forward Asia 2025 主旨演讲精彩回顾
大数据·人工智能·flink
Edingbrugh.南空1 天前
Flink ClickHouse 连接器数据读取源码深度解析
java·clickhouse·flink
Edingbrugh.南空2 天前
Flink ClickHouse 连接器维表源码深度解析
java·clickhouse·flink
Imagine Miracle2 天前
Ubuntu for ARM 更换为阿里云镜像源
arm开发·ubuntu·阿里云
draymond71073 天前
阿里云-云效自动部署spring boot项目
阿里云
诗旸的技术记录与分享3 天前
Flink-1.19.0源码详解-番外补充3-StreamGraph图
大数据·flink
Edingbrugh.南空3 天前
Flink MySQL CDC 环境配置与验证
mysql·adb·flink
我是小bā吖3 天前
阿里云服务网格ASM实践
网络·阿里云·云计算·服务发现
lxsy3 天前
spring-ai-alibaba 1.0.0.2 学习(七)——集成阿里云百炼平台知识库
学习·spring·阿里云·spring-ai·ai-alibaba
保持学习ing3 天前
苍穹外卖day3--公共字段填充+新增菜品
java·阿里云·实战·springboot·前后端·外卖项目·阿里云文件存储