FlinkSQL之保序任务对于聚合SQL影响分析

​ 本文以一个示例说明FlinkSQL如何针对上游乱序数据源设计保序任务,从而保证下游数据准确性。废话不多说,这里以交易数据场景为例.

  • 数据表结构为:

    sql 复制代码
    create table tbl_order_source(
        order_id            int             comment '订单ID',
        shop_id             int             comment '书店ID',
        user_id             int             comment '用户ID',
        original_price      double          comment '原始交易额',
        create_time         timestamp(3)    comment '创建时间: yyyy-MM-dd HH:mm:ss',
        watermark for create_time as create_time - interval '0' second
    )with(
        'connector' = 'kafka',
        'topic' = 'tbl_order_source',
        'properties.bootstrap.servers' = 'localhost:9092',
        'properties.group.id' = 'testGroup',
        'scan.startup.mode' = 'latest-offset',
        'format' = 'json',
        'json.fail-on-missing-field' = 'false',
        'json.ignore-parse-errors' = 'true'
    );
  • 乱序数据源如下:

    json 复制代码
    {"order_id":"1","shop_id":"1","user_id":"1","original_price":"1","create_time":"2024-01-01 20:05:00"}
    {"order_id":"2","shop_id":"1","user_id":"2","original_price":"2","create_time":"2024-01-01 20:04:00"}
    {"order_id":"1","shop_id":"1","user_id":"1","original_price":"3","create_time":"2024-01-01 20:03:00"}
    {"order_id":"3","shop_id":"1","user_id":"3","original_price":"4","create_time":"2024-01-01 20:02:00"}
    {"order_id":"1","shop_id":"1","user_id":"1","original_price":"5","create_time":"2024-01-01 20:04:00"}
  • 我们针对乱序数据源消费首先设计保序任务,如下:

    sql 复制代码
    -- 保序数据中间结果 
    create table ods_order_source(
        order_id            int             comment '订单ID',
        shop_id             int             comment '书店ID',
        user_id             int             comment '用户ID',
        original_price      double          comment '订单金额',
        create_time         timestamp(3)    comment '创建时间: yyyy-MM-dd HH:mm:ss',
        watermark for create_time as create_time - interval '0' second,
        primary key (order_id) not enforced
    )with(
        'connector' = 'upsert-kafka',
        'topic' = 'ods_order_source',
        'properties.bootstrap.servers' = 'localhost:9092',
        'key.format' = 'json',
        'key.json.ignore-parse-errors' = 'true',
        'value.format' = 'json',
        'value.json.fail-on-missing-field' = 'false'
    );
    
    -- 源到保序结果ETL
    insert into ods_order_source
    select
    	tmp.order_id,
    	tmp.shop_id,
    	tmp.user_id,
    	tmp.original_price,
    	tmp.create_time
    from (
    	select
    	    t.order_id,
    	    t.shop_id,
    	    t.user_id,
    	    t.original_price,
    	    t.create_time,
    		row_number()over(partition by t.order_id order by t.create_time asc) as rn
    	from tbl_order_source t
    ) tmp
    where tmp.rn = 1
    ;
    
    -- 查询保序中间结果数据
    select * from ods_order_source;

    针对数据源输入,保序任务输出为:

    json 复制代码
    +I {"order_id":"1","shop_id":"1","user_id":"1","original_price":"1","create_time":"2024-01-01 20:05:00.000"}
    +I {"order_id":"2","shop_id":"1","user_id":"2","original_price":"2","create_time":"2024-01-01 20:04:00.000"}
    -U {"order_id":"1","shop_id":"1","user_id":"1","original_price":"1","create_time":"2024-01-01 20:05:00.000"}
    +U {"order_id":"1","shop_id":"1","user_id":"1","original_price":"3","create_time":"2024-01-01 20:03:00.000"}
    +I {"order_id":"3","shop_id":"1","user_id":"3","original_price":"4","create_time":"2024-01-01 20:02:00.000"}
  • 下面针对保序任务设计聚合SQL:

    sql 复制代码
    select
        t.shop_id                                  as shop_id,
        to_date(cast(t.create_time as string))     as create_date,
        sum(t.original_price)                      as original_amt,
        sum(1)                                     as order_num,
        count(distinct t.order_id)                 as order_cnt
    from ods_order_source t
    group by
        t.shop_id,
        to_date(cast(t.create_time as string))
    ;

    测试结果:

    json 复制代码
    +I {"order_id":1,"create_date":2024-01-01,"original_amt":1,"order_num":1,"order_cnt":1}
    -U {"order_id":1,"create_date":2024-01-01,"original_amt":1,"order_num":1,"order_cnt":1}
    +U {"order_id":1,"create_date":2024-01-01,"original_amt":3,"order_num":2,"order_cnt":2}
    -U {"order_id":1,"create_date":2024-01-01,"original_amt":3,"order_num":2,"order_cnt":2}
    +U {"order_id":1,"create_date":2024-01-01,"original_amt":2,"order_num":1,"order_cnt":1}
    -U {"order_id":1,"create_date":2024-01-01,"original_amt":2,"order_num":1,"order_cnt":1}
    +U {"order_id":1,"create_date":2024-01-01,"original_amt":5,"order_num":2,"order_cnt":2}
    -U {"order_id":1,"create_date":2024-01-01,"original_amt":5,"order_num":2,"order_cnt":2}
    +U {"order_id":1,"create_date":2024-01-01,"original_amt":9,"order_num":3,"order_cnt":3}

    可以从聚合SQL输出结果看出,最后数据为{"order_id":1,"create_date":2024-01-01,"original_amt":9,"order_num":3,"order_cnt":3},结果和源输入数据对应上。

  • 下面同样的SQL针对未保序源数据做聚合,看下结果如何:

    sql 复制代码
    select
        t.shop_id                                  as shop_id,
        to_date(cast(t.create_time as string))     as create_date,
        sum(t.original_price)                      as original_amt,
        sum(1)                                     as order_num,
        count(distinct t.order_id)                 as order_cnt
    from tbl_order_source t
    group by
        t.shop_id,
        to_date(cast(t.create_time as string))
    ;

    查看结果:

    +I {"order_id":1,"create_date":2024-01-01,"original_amt":1,"order_num":1,"order_cnt":1}
    -U {"order_id":1,"create_date":2024-01-01,"original_amt":1,"order_num":1,"order_cnt":1}
    +U {"order_id":1,"create_date":2024-01-01,"original_amt":3,"order_num":2,"order_cnt":2}
    -U {"order_id":1,"create_date":2024-01-01,"original_amt":3,"order_num":2,"order_cnt":2}
    +U {"order_id":1,"create_date":2024-01-01,"original_amt":6,"order_num":3,"order_cnt":2}
    -U {"order_id":1,"create_date":2024-01-01,"original_amt":6,"order_num":3,"order_cnt":2}
    +U {"order_id":1,"create_date":2024-01-01,"original_amt":10,"order_num":4,"order_cnt":3}
    -U {"order_id":1,"create_date":2024-01-01,"original_amt":10,"order_num":4,"order_cnt":3}
    +U {"order_id":1,"create_date":2024-01-01,"original_amt":15,"order_num":4,"order_cnt":3}
    

    最后结果{"order_id":1,"create_date":2024-01-01,"original_amt":15,"order_num":4,"order_cnt":3}错误。

​ 针对测试结果,对于聚合SQL数据sink设置shop_id作为key,就可以保证下游数据结果的正确性。

相关推荐
lzhlizihang1 小时前
【Hive sql 面试题】求出各类型专利top 10申请人,以及对应的专利申请数(难)
大数据·hive·sql·面试题
Hsu_kk1 小时前
Hive 查询各类型专利 Top 10 申请人及对应的专利申请数
数据仓库·hive·hadoop
Tianyanxiao1 小时前
如何利用探商宝精准营销,抓住行业机遇——以AI技术与大数据推动企业信息精准筛选
大数据·人工智能·科技·数据分析·深度优先·零售
大数据编程之光1 小时前
Hive 查询各类型专利 top10 申请人及专利申请数
大数据·数据仓库·hive·hadoop
杰克逊的日记1 小时前
Hive详解
数据仓库·hive·hadoop
GDDGHS_1 小时前
大数据工具 flume 的安装配置与使用 (详细版)
大数据·flume
Acrelhuang2 小时前
安科瑞5G基站直流叠光监控系统-安科瑞黄安南
大数据·数据库·数据仓库·物联网
皓7413 小时前
服饰电商行业知识管理的创新实践与知识中台的重要性
大数据·人工智能·科技·数据分析·零售
Mephisto.java3 小时前
【大数据学习 | kafka高级部分】kafka的kraft集群
大数据·sql·oracle·kafka·json·hbase
Mephisto.java3 小时前
【大数据学习 | kafka高级部分】kafka的文件存储原理
大数据·sql·oracle·kafka·json