FlinkSQL之保序任务对于聚合SQL影响分析

​ 本文以一个示例说明FlinkSQL如何针对上游乱序数据源设计保序任务，从而保证下游数据准确性。废话不多说，这里以交易数据场景为例.

• 数据表结构为：

  create table tbl_order_source(
      order_id            int             comment '订单ID',
      shop_id             int             comment '书店ID',
      user_id             int             comment '用户ID',
      original_price      double          comment '原始交易额',
      create_time         timestamp(3)    comment '创建时间: yyyy-MM-dd HH:mm:ss',
      watermark for create_time as create_time - interval '0' second
  )with(
      'connector' = 'kafka',
      'topic' = 'tbl_order_source',
      'properties.bootstrap.servers' = 'localhost:9092',
      'properties.group.id' = 'testGroup',
      'scan.startup.mode' = 'latest-offset',
      'format' = 'json',
      'json.fail-on-missing-field' = 'false',
      'json.ignore-parse-errors' = 'true'
  );

• 乱序数据源如下：

  {"order_id":"1","shop_id":"1","user_id":"1","original_price":"1","create_time":"2024-01-01 20:05:00"}
  {"order_id":"2","shop_id":"1","user_id":"2","original_price":"2","create_time":"2024-01-01 20:04:00"}
  {"order_id":"1","shop_id":"1","user_id":"1","original_price":"3","create_time":"2024-01-01 20:03:00"}
  {"order_id":"3","shop_id":"1","user_id":"3","original_price":"4","create_time":"2024-01-01 20:02:00"}
  {"order_id":"1","shop_id":"1","user_id":"1","original_price":"5","create_time":"2024-01-01 20:04:00"}

• 我们针对乱序数据源消费首先设计保序任务，如下：

  -- 保序数据中间结果 
  create table ods_order_source(
      order_id            int             comment '订单ID',
      shop_id             int             comment '书店ID',
      user_id             int             comment '用户ID',
      original_price      double          comment '订单金额',
      create_time         timestamp(3)    comment '创建时间: yyyy-MM-dd HH:mm:ss',
      watermark for create_time as create_time - interval '0' second,
      primary key (order_id) not enforced
  )with(
      'connector' = 'upsert-kafka',
      'topic' = 'ods_order_source',
      'properties.bootstrap.servers' = 'localhost:9092',
      'key.format' = 'json',
      'key.json.ignore-parse-errors' = 'true',
      'value.format' = 'json',
      'value.json.fail-on-missing-field' = 'false'
  );
  
  -- 源到保序结果ETL
  insert into ods_order_source
  select
  	tmp.order_id,
  	tmp.shop_id,
  	tmp.user_id,
  	tmp.original_price,
  	tmp.create_time
  from (
  	select
  	    t.order_id,
  	    t.shop_id,
  	    t.user_id,
  	    t.original_price,
  	    t.create_time,
  		row_number()over(partition by t.order_id order by t.create_time asc) as rn
  	from tbl_order_source t
  ) tmp
  where tmp.rn = 1
  ;
  
  -- 查询保序中间结果数据
  select * from ods_order_source;

  针对数据源输入，保序任务输出为：

  +I {"order_id":"1","shop_id":"1","user_id":"1","original_price":"1","create_time":"2024-01-01 20:05:00.000"}
  +I {"order_id":"2","shop_id":"1","user_id":"2","original_price":"2","create_time":"2024-01-01 20:04:00.000"}
  -U {"order_id":"1","shop_id":"1","user_id":"1","original_price":"1","create_time":"2024-01-01 20:05:00.000"}
  +U {"order_id":"1","shop_id":"1","user_id":"1","original_price":"3","create_time":"2024-01-01 20:03:00.000"}
  +I {"order_id":"3","shop_id":"1","user_id":"3","original_price":"4","create_time":"2024-01-01 20:02:00.000"}

• 下面针对保序任务设计聚合SQL：

  select
      t.shop_id                                  as shop_id,
      to_date(cast(t.create_time as string))     as create_date,
      sum(t.original_price)                      as original_amt,
      sum(1)                                     as order_num,
      count(distinct t.order_id)                 as order_cnt
  from ods_order_source t
  group by
      t.shop_id,
      to_date(cast(t.create_time as string))
  ;

  测试结果：

  +I {"order_id":1,"create_date":2024-01-01,"original_amt":1,"order_num":1,"order_cnt":1}
  -U {"order_id":1,"create_date":2024-01-01,"original_amt":1,"order_num":1,"order_cnt":1}
  +U {"order_id":1,"create_date":2024-01-01,"original_amt":3,"order_num":2,"order_cnt":2}
  -U {"order_id":1,"create_date":2024-01-01,"original_amt":3,"order_num":2,"order_cnt":2}
  +U {"order_id":1,"create_date":2024-01-01,"original_amt":2,"order_num":1,"order_cnt":1}
  -U {"order_id":1,"create_date":2024-01-01,"original_amt":2,"order_num":1,"order_cnt":1}
  +U {"order_id":1,"create_date":2024-01-01,"original_amt":5,"order_num":2,"order_cnt":2}
  -U {"order_id":1,"create_date":2024-01-01,"original_amt":5,"order_num":2,"order_cnt":2}
  +U {"order_id":1,"create_date":2024-01-01,"original_amt":9,"order_num":3,"order_cnt":3}

  可以从聚合SQL输出结果看出，最后数据为{"order_id":1,"create_date":2024-01-01,"original_amt":9,"order_num":3,"order_cnt":3}，结果和源输入数据对应上。

• 下面同样的SQL针对未保序源数据做聚合，看下结果如何：

  select
      t.shop_id                                  as shop_id,
      to_date(cast(t.create_time as string))     as create_date,
      sum(t.original_price)                      as original_amt,
      sum(1)                                     as order_num,
      count(distinct t.order_id)                 as order_cnt
  from tbl_order_source t
  group by
      t.shop_id,
      to_date(cast(t.create_time as string))
  ;

  查看结果：

  +I {"order_id":1,"create_date":2024-01-01,"original_amt":1,"order_num":1,"order_cnt":1}
  -U {"order_id":1,"create_date":2024-01-01,"original_amt":1,"order_num":1,"order_cnt":1}
  +U {"order_id":1,"create_date":2024-01-01,"original_amt":3,"order_num":2,"order_cnt":2}
  -U {"order_id":1,"create_date":2024-01-01,"original_amt":3,"order_num":2,"order_cnt":2}
  +U {"order_id":1,"create_date":2024-01-01,"original_amt":6,"order_num":3,"order_cnt":2}
  -U {"order_id":1,"create_date":2024-01-01,"original_amt":6,"order_num":3,"order_cnt":2}
  +U {"order_id":1,"create_date":2024-01-01,"original_amt":10,"order_num":4,"order_cnt":3}
  -U {"order_id":1,"create_date":2024-01-01,"original_amt":10,"order_num":4,"order_cnt":3}
  +U {"order_id":1,"create_date":2024-01-01,"original_amt":15,"order_num":4,"order_cnt":3}

  最后结果{"order_id":1,"create_date":2024-01-01,"original_amt":15,"order_num":4,"order_cnt":3}错误。

​ 针对测试结果，对于聚合SQL数据sink设置shop_id作为key，就可以保证下游数据结果的正确性。