目录
[一、FlinkSQL 的窗口操作](#一、FlinkSQL 的窗口操作)
[二、窗口 TopN 案例解析](#二、窗口 TopN 案例解析)
在大数据处理领域,Flink 凭借其卓越的流批一体处理能力、高效的状态管理以及精准的时间语义把控,成为众多开发者应对复杂数据场景的得力工具。本文将围绕 Flink 的三大关键板块展开深度剖析,即 FlinkSQL 的窗口操作(涵盖滚动、滑动、累积窗口以及不同时间语义下的应用)、窗口 TopN 需求实现案例,以及 Flink 架构体系(详细解读各个核心组件及概念),旨在为读者清晰勾勒 Flink 技术框架的全貌,助力深入理解与高效运用。
一、FlinkSQL 的窗口操作
(一)窗口类型概述
滚动窗口(Tumble)
滚动窗口特点鲜明,它将数据依据固定的时间长度切分成一个个互不重叠的 "窗口片段"。例如,设定滚动窗口大小为 1 小时,那么数据会按照每小时为单位进行规整统计,前一小时的数据汇聚在一个窗口内处理,下一小时数据则落入后续独立窗口,彼此界限清晰,无交叉重叠区域,适用于对固定周期数据做独立聚合分析场景,像按日统计网站每日访问量等。
滑动窗口(HOP)
滑动窗口相对灵活,它有两个关键参数:窗口大小和滑动步长。窗口大小界定了数据统计的时间跨度,滑动步长则规定了窗口每次移动的时间间隔。假设窗口大小设为 1 小时,滑动步长设为 10 分钟,意味着每隔 10 分钟就会生成一个覆盖过去 1 小时数据的窗口进行统计分析,窗口之间会有重叠部分,利于捕捉数据在短时间内连续变化趋势,常用于监控系统实时指标统计,如近 1 小时内每隔 10 分钟统计系统平均负载。
累积窗口(Cumulate)
累积窗口专为长周期统计且需中间多次输出累积结果场景设计。设有最大窗口长度(统计周期)与累积步长两核心参数,起始窗口大小为累积步长,后续窗口依次在前一窗口基础上按步长拓展,直至达最大窗口长度。如统计电商平台一周内每日累计订单金额,可设最大窗口长度 7 天,累积步长 1 天,每天输出当前累计订单金额数据,呈现数据逐步累加态势。
(二)不同时间语义下窗口实践
EventTime(事件时间)
基于事件实际发生时间处理数据,契合数据产生源头逻辑顺序,但使用时需重点关注水印设置。以统计用户消费金额场景为例,测试数据含用户消费记录及对应事件时间,创建表时要将时间字段设为 TIMESTAMP (3) 类型,并添加水印(如 watermark for event_time as event_time - interval '3' second
)用于处理乱序、延迟数据,确保窗口计算精准。
测试数据如下:
java
{"username":"zs","price":20,"event_time":"2023-07-17 10:10:10"}
{"username":"zs","price":15,"event_time":"2023-07-17 10:10:30"}
{"username":"zs","price":20,"event_time":"2023-07-17 10:10:40"}
{"username":"zs","price":20,"event_time":"2023-07-17 10:11:03"}
{"username":"zs","price":20,"event_time":"2023-07-17 10:11:04"}
{"username":"zs","price":20,"event_time":"2023-07-17 10:12:04"}
{"username":"zs","price":20,"event_time":"2023-07-17 11:12:04"}
{"username":"zs","price":20,"event_time":"2023-07-17 11:12:04"}
{"username":"zs","price":20,"event_time":"2023-07-17 12:12:04"}
{"username":"zs","price":20,"event_time":"2023-07-18 12:12:04"}
需求:每隔1分钟统计这1分钟的每个用户的总消费金额和消费次数
需要用到滚动窗口
编写好sql:
sql
CREATE TABLE table1 (
`username` string,
`price` int,
`event_time` TIMESTAMP(3),
watermark for event_time as event_time - interval '3' second
) WITH (
'connector' = 'kafka',
'topic' = 'topic1',
'properties.bootstrap.servers' = 'bigdata01:9092',
'properties.group.id' = 'g1',
'scan.startup.mode' = 'latest-offset',
'format' = 'json'
);
select
window_start,
window_end,
username,
count(1) zongNum,
sum(price) totalMoney
from table(TUMBLE(TABLE table1, DESCRIPTOR(event_time), INTERVAL '60' second))
group by window_start,window_end,username;
滚动窗口示例:
java
package com.bigdata.day08;
import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
public class _03EventTimeGunDongWindowDemo {
public static void main(String[] args) throws Exception {
//1. env-准备环境
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1);
StreamTableEnvironment tenv = StreamTableEnvironment.create(env);
//2. 创建表
tenv.executeSql("CREATE TABLE table1 (\n" +
" `username` String,\n" +
" `price` int,\n" +
" `event_time` TIMESTAMP(3),\n" +
" watermark for event_time as event_time - interval '3' second\n" +
") WITH (\n" +
" 'connector' = 'kafka',\n" +
" 'topic' = 'topic1',\n" +
" 'properties.bootstrap.servers' = 'bigdata01:9092',\n" +
" 'properties.group.id' = 'testGroup1',\n" +
" 'scan.startup.mode' = 'group-offsets',\n" +
" 'format' = 'json'\n" +
")");
//3. 通过sql语句统计结果
tenv.executeSql("select \n" +
" window_start,\n" +
" window_end,\n" +
" username,\n" +
" count(1) zongNum,\n" +
" sum(price) totalMoney \n" +
" from table(TUMBLE(TABLE table1, DESCRIPTOR(event_time), INTERVAL '60' second))\n" +
"group by window_start,window_end,username").print();
//4. sink-数据输出
//5. execute-执行
env.execute();
}
}
统计结果如下:
滑动窗口示例:
java
package com.bigdata.day08;
import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
public class _03EventTimeGunDongWindowDemo {
public static void main(String[] args) throws Exception {
//1. env-准备环境
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1);
StreamTableEnvironment tenv = StreamTableEnvironment.create(env);
//2. 创建表
tenv.executeSql("CREATE TABLE table1 (\n" +
" `username` String,\n" +
" `price` int,\n" +
" `event_time` TIMESTAMP(3),\n" +
" watermark for event_time as event_time - interval '3' second\n" +
") WITH (\n" +
" 'connector' = 'kafka',\n" +
" 'topic' = 'topic1',\n" +
" 'properties.bootstrap.servers' = 'bigdata01:9092',\n" +
" 'properties.group.id' = 'testGroup1',\n" +
" 'scan.startup.mode' = 'group-offsets',\n" +
" 'format' = 'json'\n" +
")");
//3. 通过sql语句统计结果
tenv.executeSql("select \n" +
" window_start,\n" +
" window_end,\n" +
" username,\n" +
" count(1) zongNum,\n" +
" sum(price) totalMoney \n" +
" from table(HOP(TABLE table1, DESCRIPTOR(event_time), INTERVAL '10' second,INTERVAL '60' second))\n" +
"group by window_start,window_end,username").print();
//4. sink-数据输出
//5. execute-执行
env.execute();
}
}
结果如图所示:
累积窗口示例:
java
package com.bigdata.day08;
import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
public class _03EventTimeGunDongWindowDemo {
public static void main(String[] args) throws Exception {
//1. env-准备环境
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1);
StreamTableEnvironment tenv = StreamTableEnvironment.create(env);
//2. 创建表
tenv.executeSql("CREATE TABLE table1 (\n" +
" `username` String,\n" +
" `price` int,\n" +
" `event_time` TIMESTAMP(3),\n" +
" watermark for event_time as event_time - interval '3' second\n" +
") WITH (\n" +
" 'connector' = 'kafka',\n" +
" 'topic' = 'topic1',\n" +
" 'properties.bootstrap.servers' = 'bigdata01:9092',\n" +
" 'properties.group.id' = 'testGroup1',\n" +
" 'scan.startup.mode' = 'group-offsets',\n" +
" 'format' = 'json'\n" +
")");
//3. 通过sql语句统计结果
tenv.executeSql("select \n" +
" window_start,\n" +
" window_end,\n" +
" username,\n" +
" count(1) zongNum,\n" +
" sum(price) totalMoney \n" +
" from table(CUMULATE(TABLE table1, DESCRIPTOR(event_time), INTERVAL '1' hours,INTERVAL '1' days))\n" +
"group by window_start,window_end,username").print();
//4. sink-数据输出
//5. execute-执行
env.execute();
}
}
累积窗口演示效果:
ProcessTime(处理时间)
依数据进入 Flink 系统并被处理当下时间为准,简单直接,无需水印设置,但易受系统处理速度、网络延迟等因素干扰,数据顺序未必契合事件真实发生先后。如:
测试数据:
java
{"username":"zs","price":20}
{"username":"lisi","price":15}
{"username":"lisi","price":20}
{"username":"zs","price":20}
{"username":"zs","price":20}
{"username":"zs","price":20}
{"username":"zs","price":20}
java
/**
* 滚动窗口大小1分钟 延迟时间3秒
*
* {"username":"zs","price":20}
* {"username":"lisi","price":15}
* {"username":"lisi","price":20}
* {"username":"zs","price":20}
* {"username":"zs","price":20}
* {"username":"zs","price":20}
* {"username":"zs","price":20}
*
*/
package com.bigdata.day08;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
public class _04ProcessingTimeGunDongWindowDemo {
public static void main(String[] args) throws Exception {
//1. env-准备环境
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1);
StreamTableEnvironment tenv = StreamTableEnvironment.create(env);
//2. 创建表
tenv.executeSql("CREATE TABLE table1 (\n" +
" `username` String,\n" +
" `price` int,\n" +
" `event_time` as proctime()\n" +
") WITH (\n" +
" 'connector' = 'kafka',\n" +
" 'topic' = 'topic1',\n" +
" 'properties.bootstrap.servers' = 'bigdata01:9092',\n" +
" 'properties.group.id' = 'testGroup1',\n" +
" 'scan.startup.mode' = 'group-offsets',\n" +
" 'format' = 'json'\n" +
")");
//3. 通过sql语句统计结果
tenv.executeSql("select \n" +
" window_start,\n" +
" window_end,\n" +
" username,\n" +
" count(1) zongNum,\n" +
" sum(price) totalMoney \n" +
" from table(TUMBLE(TABLE table1, DESCRIPTOR(event_time), INTERVAL '60' second ))\n" +
"group by window_start,window_end,username").print();
//4. sink-数据输出
//5. execute-执行
env.execute();
}
}
计算结果:
结果需要等1分钟,才能显示出来,不要着急!
窗口分为滚动和滑动,时间分为事件时间和处理时间,两两组合,4个案例。
以下是滑动窗口+处理时间:
java
package com.bigdata.sql;
import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
public class _04_FlinkSQLProcessTime_HOP {
public static void main(String[] args) throws Exception {
//1. env-准备环境
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
// 获取tableEnv对象
// 通过env 获取一个table 环境
StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
tEnv.executeSql("CREATE TABLE table1 (\n" +
" `username` string,\n" +
" `price` int,\n" +
" `event_time` as proctime() \n"+
") WITH (\n" +
" 'connector' = 'kafka',\n" +
" 'topic' = 'topic1',\n" +
" 'properties.bootstrap.servers' = 'bigdata01:9092',\n" +
" 'properties.group.id' = 'g1',\n" +
" 'scan.startup.mode' = 'latest-offset',\n" +
" 'format' = 'json'\n" +
")");
// 语句中的 ; 不能添加
tEnv.executeSql("select \n" +
" window_start,\n" +
" window_end,\n" +
" username,\n" +
" count(1) zongNum,\n" +
" sum(price) totalMoney \n" +
" from table(HOP(TABLE table1, DESCRIPTOR(event_time),INTERVAL '10' second, INTERVAL '60' second))\n" +
"group by window_start,window_end,username").print();
//5. execute-执行
env.execute();
}
}
测试时假如你的控制台不出数据,触发不了,请进入如下操作:
1、重新创建一个新的 topic,分区数为 1
2、kafka 对接的 server,写全 bigdata01:9092,bigdata02:9092,bigdata03:9092
二、窗口 TopN 案例解析
需求背景
在海量数据场景下,常需在特定时间窗口内筛选出具备突出特征数据,如找出每个小时内点击量最多的 Top 3 网页,洞察热门页面趋势助力优化运营策略。
测试数据
bash
{"ts": "2023-09-05 12:00:00", "page_id": 1, "clicks": 100}
{"ts": "2023-09-05 12:01:00", "page_id": 2, "clicks": 90}
{"ts": "2023-09-05 12:10:00", "page_id": 3, "clicks": 110}
{"ts": "2023-09-05 12:20:00", "page_id": 4, "clicks": 23}
{"ts": "2023-09-05 12:30:00", "page_id": 5, "clicks": 456}
{"ts": "2023-09-05 13:10:00", "page_id": 5, "clicks": 456}
基础统计逻辑
先基于滚动窗口统计各网页在每小时内点击量总和,利用 tumble
窗口按 1 小时(INTERVAL '1' HOUR
)切分数据,以网页 page_id
分组聚合点击量 clicks
,代码如下:
sql
假如没有每隔1小时的需求,仅仅是统计点击量最多的Top 3网页,结果如下
select * from (
select
page_id,
totalSum,
row_number() over (order by totalSum desc) px
from (
select page_id,
sum(clicks) totalSum
from kafka_page_clicks group by page_id ) ) where px <=3;
根据以上代码,添加滚动窗口的写法:
sql
select
window_start,
window_end,
page_id,
sum(clicks) totalSum
from
table (
tumble( table kafka_page_clicks, descriptor(ts), INTERVAL '1' HOUR )
)
group by window_start,window_end,page_id;
添加排名逻辑:
借助窗口函数 row_number() over(partition by window_start,window_end order by totalSum desc )
对各窗口内网页按点击量降序排名,筛选排名前 3(where pm <= 3
)记录,完整代码如下:
sql
select
window_start,
window_end,
page_id,
pm
from (
select
window_start,
window_end,
page_id,
row_number() over(partition by window_start,window_end order by totalSum desc ) pm
from (
select
window_start,
window_end,
page_id,
sum(clicks) totalSum
from
table (
tumble( table kafka_page_clicks, descriptor(ts), INTERVAL '1' HOUR )
)
group by window_start,window_end,page_id ) t2 ) t1 where pm <= 3;
编写建表语句:
java
{"ts": "2023-09-05 12:00:00", "page_id": 1, "clicks": 100}
CREATE TABLE kafka_page_clicks (
`ts` TIMESTAMP(3),
`page_id` int,
`clicks` int,
watermark for ts as ts - interval '3' second
) WITH (
'connector' = 'kafka',
'topic' = 'topic1',
'properties.bootstrap.servers' = 'bigdata01:9092',
'properties.group.id' = 'g1',
'scan.startup.mode' = 'latest-offset',
'format' = 'json'
)
java
package com.bigdata.day08;
import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
public class _05TopNDemo {
public static void main(String[] args) throws Exception {
//1. env-准备环境
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// ctrl + y 删除光标所在的那一行数据 ctrl + d 复制当前行
StreamTableEnvironment tenv = StreamTableEnvironment.create(env);
//2. source-加载数据
// 一定要注意:ts 是一个年月日时分秒的数据,所以在建表时一定要是TIMESTAMP,否则进行WATERMARK 报错
// 因为使用的是event_time 所以,需要指定WATERMARK
tenv.executeSql("CREATE TABLE kafka_page_clicks (" +
" `ts` TIMESTAMP(3),\n" +
" page_id INT,\n" +
" clicks INT,\n" +
" WATERMARK FOR ts AS ts - INTERVAL '10' SECOND \n" +
") WITH (\n" +
" 'connector' = 'kafka',\n" +
" 'topic' = 'topic1',\n" +
" 'properties.bootstrap.servers' = 'bigdata01:9092',\n" +
" 'scan.startup.mode' = 'group-offsets',\n" +
" 'format' = 'json'\n" +
")");
tenv.executeSql("select \n" +
" window_start,\n" +
" window_end,\n" +
" page_id,\n" +
" pm\n" +
" from (\n" +
"select \n" +
" window_start,\n" +
" window_end,\n" +
" page_id,\n" +
" row_number() over(partition by window_start,window_end order by totalSum desc ) pm\n" +
" from (\n" +
"select \n" +
" window_start,\n" +
" window_end,\n" +
" page_id,\n" +
" sum(clicks) totalSum \n" +
" from \n" +
" table ( \n" +
" tumble( table kafka_page_clicks, descriptor(ts), INTERVAL '1' HOUR ) \n" +
" ) \n" +
" group by window_start,window_end,page_id ) t2 ) t1 where pm <= 3").print();
//4. sink-数据输出
//5. execute-执行
env.execute();
}
}
最后的运行结果如下:
三、Flink 架构体系
(一)核心组件
JobManager(Master)
作为 Flink 集群 "指挥官",负责协调分布式任务执行全流程。承担调度任务(依据资源与任务优先级合理分配执行)、协调检查点(保障数据一致性与容错恢复关键机制,定期触发全局或局部快照)、故障恢复(集群节点故障时依检查点重启并恢复任务状态)等核心职责。高可用配置下可有多个实例,其中一个为 leader 主导操作,其余 standby 随时待命接管。
TaskManager(Worker)
扮演 "实干者" 角色,执行具体数据处理任务,负责运行 dataflow 的 task 或 subtask,提供数据缓冲空间优化数据读写,以及保障 data stream 在不同 task 间高效交换。集群至少需一个 TaskManager 实例,依据业务负载与资源可横向扩展提升处理能力。
Slot(任务执行槽位)
物理概念,一个TM(TaskManager)内会划分出多个Slot,1个Slot内最多可以运行1个Task(Subtask)或一组由Task(Subtask)组成的任务链。(类似于 Container)
多个Slot之间会共享平分当前TM的内存空间。Slot是对一个TM的资源进行固定分配的工具,每个Slot在TM启动后,可以获得固定的资源。比如1个TM是一个JVM进程,如果有6个Slot,那么这6个Slot平分这一个JVM进程的资源,但是因为在同一个进程内,所以线程之间共享TCP连接、内存数据等,效率更高(Slot之间交流方便)
(二)关键概念
Task 与 Subtask
一个 Flink 作业(Job)依并行度、算子类型拆解成多个 Task,Task 是作业执行基本单元;而 Subtask 是 Task 并行执行细分,Task 并行度决定其 Subtask 数量,如并行度设为 8,对应 Task 便有 8 个 Subtask 并行运作,恰似多线程协同处理任务提升整体效率。
并行度
并行度就是一个Task可以分成多少个Subtask并行执行的一个参数。这个参数是动态的,可以在任务执行前进行分配,而非Slot分配,TM启动就固定了。
一个Task可以获得的最大并行度取决于整个Flink环境的可用Slot数量,也就是如果有8个Slot,那么最大并行度也就是8,设置的再大也没有意义(还报错)。
假如你只有6个槽,并行度设置为8,启动一会儿之后会报错,启动任务失败,报错如下:
集群中槽的数量虽然是手动设置的,但是也不能超过集群中的 CPU 总核数。
如下图:
- 一个Job分为了3个Task来运行,分别是TaskA TaskB TaskC
- 其中TaskA设置为了6个并行度,也就是TaskA可以有6个Subtask,如图可见,TaskA的6个Subtask各自在一个Slot内执行
- 其中在Slot的时候说过,Slot可以运行由Task(或Subtask)组成的任务链,如图可见,最左边的Slot运行了TaskA TaskB TaskC 3个Task各自的1个Subtask组成的一个Subtask执行链
并行度是一个动态的概念,可以在多个地方设置并行度:【重要】
- 配置文件默认并行度:conf/flink-conf.yaml的parallelism.default
- 启动Flink任务,动态提交参数:比如:bin/flink run -p 3 xxx.jar
- 在代码中设置全局并行度:env.setParallelism(3);
- 针对每个算子进行单独设置:sum(1).setParallelism(3)
优先级:算子 > 代码全局 > 命令行参数 > 配置文件
四、总结
本文对 FlinkSQL 窗口操作多类型应用、窗口 TopN 经典案例实现细节,以及 Flink 架构体系底层逻辑展开全方位解读。掌握窗口操作可灵活应对多样时间序列数据统计分析;理解架构体系助于合理规划集群资源、高效编排任务执行流程,期望为开发者驾驭 Flink 处理大数据难题筑牢根基、开拓思路。后续可深入探索 Flink 状态管理、与外部存储交互等进阶主题,深挖其技术潜能。