flink命令行提交jar包任务

1. 环境准备

1.1 flink环境准备

关于如何安装flink，这个写的非常详细，https://blog.csdn.net/qq_43699958/article/details/132826440

在flink的bin目录启动flink cluster

[root@localhost bin]# ./start-cluster.sh

1.2 Linux环境准备

1.2.1 关闭linux防火墙

会用到的命令如下，

a. 查看防火墙状态：firewall-cmd --state

如果是not running状态，说明没有启动防火墙

b. 关闭防火墙：systemctl stop firewalld.service

c. 设置开机禁启：systemctl disable firewalld.service

[root@localhost bin]# firewall-cmd --state
not running
[root@localhost bin]# systemctl stop firewalld.service
[root@localhost bin]# systemctl disable firewalld.service
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@localhost bin]#

1.2.2 hosts文件准备

因为下文的java代码中，代码段socketTextStream("hadoop01", 8888)涉及到了host名称，所以需要将linux环境中的/ect/hosts修改一下

文件中添加上这么一行

192.168.126.223 hadoop01

1.3 nc准备

启动一个数据源，用到命令为

[root@localhost ~]# nc -lk 8888

上面命令的意思是：

-k, --keep-open Accept multiple connections in listen mode

-l, --listen Bind and listen for incoming connections

1.4 jar包准备

代码如下，打包成一个jar包即可

package com.atguigu.wc;

import org.apache.flink.api.common.typeinfo.Types;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

/**
 * @author Amos
 * @date 2023/9/11
 */

public class WordCountStreamUnboundedDemo {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStreamSource<String> socketDS = env.socketTextStream("hadoop01", 8888);
        SingleOutputStreamOperator<Tuple2<String, Integer>> sum = socketDS
                .flatMap(
                        (String value, Collector<Tuple2<String, Integer>> out) -> {
                            String[] words = value.split(" ");
                            for (String word : words) {
                                out.collect(Tuple2.of(word, 1));
                            }
                        }
                )
                .returns(Types.TUPLE(Types.STRING, Types.INT))
                .keyBy(value -> value.f0)
                .sum(1);

        sum.print();
        env.execute();
    }
}

2. jar包提交

2.1 首先打开nc

[root@localhost ~]# nc -lk 8888

2.2 在flink环境提交jar包

为了方便，直接将jar包放到了bin目录中，名称为上面java代码的工程名称，

MyFlinkTutorial-117-1.0-SNAPSHOT.jar

[root@localhost bin]# pwd
/root/flink-1.17.1/bin
[root@localhost bin]# ll
total 2364
-rw-r--r--. 1  501 games 2290658 May 19 07:13 bash-java-utils.jar
-rwxr-xr-x. 1  501 games   22811 May 19 04:43 config.sh
-rwxr-xr-x. 1  501 games    1318 May 19 04:43 find-flink-home.sh
-rwxr-xr-x. 1  501 games    2381 May 19 04:43 flink
-rwxr-xr-x. 1  501 games    4446 May 19 04:43 flink-console.sh
-rwxr-xr-x. 1  501 games    6783 May 19 04:43 flink-daemon.sh
-rwxr-xr-x. 1  501 games    1564 May 19 04:43 historyserver.sh
-rwxr-xr-x. 1  501 games    2498 May 19 04:43 jobmanager.sh
-rwxr-xr-x. 1  501 games    1650 May 19 04:43 kubernetes-jobmanager.sh
-rwxr-xr-x. 1  501 games    1717 May 19 04:43 kubernetes-session.sh
-rwxr-xr-x. 1  501 games    1770 May 19 04:43 kubernetes-taskmanager.sh
-rw-r--r--. 1 root root     9055 Sep 12 03:41 MyFlinkTutorial-117-1.0-SNAPSHOT.jar
-rwxr-xr-x. 1  501 games    2994 May 19 04:43 pyflink-shell.sh
-rwxr-xr-x. 1  501 games    4051 May 19 04:44 sql-client.sh
-rwxr-xr-x. 1  501 games    3299 May 19 04:44 sql-gateway.sh
-rwxr-xr-x. 1  501 games    2006 May 19 04:43 standalone-job.sh
-rwxr-xr-x. 1  501 games    1837 May 19 04:43 start-cluster.sh
-rwxr-xr-x. 1  501 games    1854 May 19 04:43 start-zookeeper-quorum.sh
-rwxr-xr-x. 1  501 games    1617 May 19 04:43 stop-cluster.sh
-rwxr-xr-x. 1  501 games    1845 May 19 04:43 stop-zookeeper-quorum.sh
-rwxr-xr-x. 1  501 games    2960 May 19 04:43 taskmanager.sh
-rwxr-xr-x. 1  501 games    1725 May 19 04:43 yarn-session.sh
-rwxr-xr-x. 1  501 games    2405 May 19 04:43 zookeeper.sh
[root@localhost bin]#

因为1.1已经启动了flink环境，并且2.1中启动了nc，所以这里直接提交jar包

命令为：

./flink run -c com.atguigu.wc.WordCountStreamUnboundedDemo MyFlinkTutorial-117-1.0-SNAPSHOT.jar

命令中，-c 指定了包的入口类为com.atguigu.wc.WordCountStreamUnboundedDemo， 后面接上jar名称即可

然后会生成一个jobid

[root@localhost bin]# ./flink run -c com.atguigu.wc.WordCountStreamUnboundedDemo MyFlinkTutorial-117-1.0-SNAPSHOT.jar
Job has been submitted with JobID 28526936241a9f2486c15c9ddb7faa64

job正常提交和运行，具体如何配置webui，可以看这个文章，也是我写的，所以很有连贯性，https://blog.csdn.net/qq_43699958/article/details/132826440

nc中输入单词测试

[root@localhost ~]# nc -lk 8888
google flink
haerbin
tangshan
hangzhou
kunming
留下青春的都
jiayou nikeyi

搞定！