Spark SQL自定义collect_list分组排序

想要在spark sql中对group by + concat_ws()的字段进行排序,可以参考如下方法。

原始数据如下:

java 复制代码
+---+-----+----+
|id |name |type|
+---+-----+----+
|1  |name1|p   |
|2  |name2|p   |
|3  |name3|p   |
|1  |x1   |q   |
|2  |x2   |q   |
|3  |x3   |q   |
+---+-----+----+

目标数据如下:

java 复制代码
+----+---------------------+
|type|value_list           |
+----+---------------------+
|p   |[name3, name2, name1]|
|q   |[x3, x2, x1]         |
+----+---------------------+

spark-shell:

java 复制代码
val df=Seq((1,"name1","p"),(2,"name2","p"),(3,"name3","p"),(1,"x1","q"),(2,"x2","q"),(3,"x3","q")).toDF("id","name","type")
df.show(false)

1.使用开窗函数

java 复制代码
df.createOrReplaceTempView("test")
spark.sql("select type,max(c) as c1 from (select type,concat_ws('&',collect_list(trim(name)) over(partition by type order by id desc)) as c  from test) as x group by type ")

因为使用开窗函数本身会使用比较多的资源,

这种方式在大数据量下性能会比较慢,所以尝试下面的操作。

2.使用struct和sort_array(array,asc?true,flase)的方式来进行,效率高些:

java 复制代码
val df3=spark.sql("select type, concat_ws('&',sort_array(collect_list(struct(id,name)),false).name) as c from test group by type ")
df3.show(false)

例如:计算一个结果形如:

java 复制代码
user_id    stk_id:action_type:amount:price:time   stk_id:action_type:amount:price:time   stk_id:action_type:amount:price:time   stk_id:action_type:amount:price:time 

需要按照time 升序排,则:

java 复制代码
Dataset<Row> splitStkView = session.sql("select client_id, innercode, entrust_bs, business_amount, business_price, trade_date from\n" +
                "(select client_id,\n" +
                "       split(action,':')[0] as innercode,\n" +
                "       split(action,':')[1] as entrust_bs,\n" +
                "       split(action,':')[2] as business_amount,\n" +
                "       split(action,':')[3] as business_price,\n" +
                "       split(action,':')[4] as trade_date,\n" +
                "       ROW_NUMBER() OVER(PARTITION BY split(action,':')[0] ORDER BY split(action,':')[4] DESC) AS rn\n" +
                "from stk_temp)\n" +
                "where rn <= 5000");
        splitStkView.createOrReplaceTempView("splitStkView");
        Dataset<Row> groupStkView = session.sql("select client_id, CONCAT(innercode, ':', entrust_bs, ':', business_amount, ':', business_price, ':', trade_date) as behive, trade_date from splitStkView");
        groupStkView.createOrReplaceTempView("groupStkView");
        Dataset<Row> resultData = session.sql("SELECT client_id, concat_ws('\t',sort_array(collect_list(struct(trade_date, behive)),true).behive) as behives FROM groupStkView GROUP BY client_id");
        

3.udf的方式

java 复制代码
import org.apache.spark.sql.functions._
import org.apache.spark.sql._
val sortUdf = udf((rows: Seq[Row]) => {
  rows.map { case Row(id:Int, value:String) => (id, value) }
    .sortBy { case (id, value) => -id } //id if asc
    .map { case (id, value) => value }
})

val grouped = df.groupBy(col("type")).agg(collect_list(struct("id", "name")) as "id_name")
val r1 = grouped.select(col("type"), sortUdf(col("id_name")).alias("value_list"))
r1.show(false)
相关推荐
Decay__11 小时前
SQLServer 运维自动化平台 —— PowerShell + SQL Agent 作业:部署、监控、巡检、告警、工单闭环一体化
运维·数据库·sql·mysql·架构
星恒随风12 小时前
MySQL 从零入门(十):MVCC、Undo Log 与 Read View 详解——InnoDB 如何实现事务隔离?
linux·数据库·笔记·sql·mysql
Decay__17 小时前
附 - 02・SQLServer 分库分表与中间件 —— 原生分区表、横向分库实践与联邦查询
运维·数据库·sql·mysql·架构
wo_jack19 小时前
C++ 入门笔记(十八):list 进阶 —— const 迭代器、operator-> 与 clear 的坑
数据结构·c++·面试·stl·list
Decay__1 天前
SQLServer 优化案例集 —— 慢 SQL 优化、索引重构与参数调整收益对比
运维·数据库·sql·mysql·架构
2601_967980112 天前
python list如何赋值
python·list·列表推导式·赋值·切片赋值
codeejun3 天前
每日一Go·MySQL-15、慢查询日志实战分析(从 0 到 1 实战排查慢 SQL)
sql·mysql·golang
Decay__3 天前
SQLServer 监控体系构建 —— sql_exporter + Prometheus + Grafana 全栈实战
运维·数据库·sql·mysql·架构
starzy19904 天前
Structured Streaming 结构化流概述:把流数据当作一张无限增长的表
大数据·spark
Pniubi4 天前
怎么排查慢SQL,怎么解决?
java·数据库·sql