flink StreamGraph 构造flink任务

文章目录

背景

通常使用flink 提供的高级算子来编写flink 任务,对底层不是很了解,尤其是如何生成作业图的细节

下面通过构造一个有向无环图,来实际看一下

主要步骤

1.增加source

2.增加operator

  1. 增加一条边,连接source和operator

  2. 增加sink

  3. 增加一条边,连接operator和sink

代码

bash 复制代码
 // Step 1: Create basic configurations
        Configuration configuration = new Configuration();
        ExecutionConfig executionConfig = new ExecutionConfig();
        CheckpointConfig checkpointConfig = new CheckpointConfig();
        SavepointRestoreSettings savepointRestoreSettings = SavepointRestoreSettings.none();

        // Step 2: Create a new StreamGraph instance
        StreamGraph streamGraph = new StreamGraph(configuration, executionConfig, checkpointConfig, savepointRestoreSettings);

        // Step 3: Add a source operator

        GeneratorFunction<Long, String> generatorFunction = index -> "Number: " + index;
        DataGeneratorSource<String> source = new DataGeneratorSource<>(generatorFunction, Long.MAX_VALUE, RateLimiterStrategy.perSecond(1), Types.STRING);
        SourceOperatorFactory<String> sourceOperatorFactory = new SourceOperatorFactory<>(source, WatermarkStrategy.noWatermarks());
        streamGraph.addSource(1, "sourceNode", "sourceDescription", sourceOperatorFactory, TypeInformation.of(String.class), TypeInformation.of(String.class), "sourceSlot");

        // Step 4: Add a map operator to transform the data
        StreamMap<String, String> mapOperator = new StreamMap<>(new MapFunction<String, String>() {
            @Override
            public String map(String value) throws Exception {
                return value;
            }
        });
        SimpleOperatorFactory<String> mapOperatorFactory = SimpleOperatorFactory.of(mapOperator);
        streamGraph.addOperator(2, "mapNode", "mapDescription", mapOperatorFactory, TypeInformation.of(String.class), TypeInformation.of(String.class), "mapSlot");

        // Step 5: Connect source and map operator
        streamGraph.addEdge(1, 2, 0);

        // Step 6: Add a sink operator to consume the data
        StreamMap<String, String> sinkOperator = new StreamMap<>(new MapFunction<String, String>() {
            @Override
            public String map(String value) throws Exception {
                System.out.println(value);
                return value;
            }
        });
        SimpleOperatorFactory<String> sinkOperatorFactory = SimpleOperatorFactory.of(sinkOperator);
        streamGraph.addSink(3, "sinkNode", "sinkDescription", sinkOperatorFactory, TypeInformation.of(String.class), TypeInformation.of(String.class), "sinkSlot");

        // Step 7: Connect map and sink operator
        streamGraph.addEdge(2, 3, 0);
        streamGraph.setTimeCharacteristic(TimeCharacteristic.ProcessingTime);
        streamGraph.setMaxParallelism(1,1);
        streamGraph.setMaxParallelism(2,1);
        streamGraph.setMaxParallelism(3,1);
        streamGraph.setGlobalStreamExchangeMode(GlobalStreamExchangeMode.ALL_EDGES_PIPELINED);


        // Step 8: Convert StreamGraph to JobGraph
        JobGraph jobGraph = streamGraph.getJobGraph();


        // Step 9: Set up a MiniCluster for local execution
        MiniClusterConfiguration miniClusterConfig = new MiniClusterConfiguration.Builder()
                .setNumTaskManagers(10)
                .setNumSlotsPerTaskManager(10)
                .build();
        MiniCluster miniCluster = new MiniCluster(miniClusterConfig);

        // Step 10: Start the MiniCluster
        miniCluster.start();

        // Step 11: Submit the job to the MiniCluster
        JobExecutionResult result = miniCluster.executeJobBlocking(jobGraph);
        System.out.println("Job completed with result: " + result);

        // Step 12: Stop the MiniCluster
        miniCluster.close();
相关推荐
大大大大晴天12 小时前
每天认识一个组件:VictoriaMetrics
大数据
龙腾AI白云13 小时前
具身智能的空间导航与未知环境探索
大数据·数据库·人工智能·机器学习·知识图谱
演艺大赏14 小时前
三个坑连环套:一次 NameNode 起不来,从 JournalNode 未格式化查到配置里多了个挂载点
大数据·hadoop
Aloudata15 小时前
服饰零售数据治理如何统一款色码与库存口径,支撑门店售罄率和补货分析?
大数据·数据分析·零售
躺柒15 小时前
读数据分析思维通识课07数据思维图
大数据·数据挖掘·数据分析·大数据分析·数据思维图
回眸&啤酒鸭17 小时前
【回眸】Hy4 模型效果展示与能力评测大纲
大数据·人工智能·机器学习
伞伞悦读18 小时前
【第86期】分类指标与错误成本对照的实验与排查
大数据·人工智能·机器学习
金融小师妹1 天前
多因子关联模型:黄金与铜价同步回落背后的金融变量及PPI传导分析
大数据·人工智能·金融
TDengine (老段)1 天前
TDengine IDMP 功能点—数据接入
大数据·数据库·物联网·时序数据库·tdengine·涛思数据
bksczm1 天前
InnoDB 通用表空间与临时表空间详解
大数据·数据库·mysql