Hive/Spark窗口函数

有数的编程笔记2025-08-24 16:09

1. Hive和Spark SQL中窗口函数

一个完整的窗口函数包含3部分

  • 聚合函数或专用窗口函数
  • over()子句
  • 窗口规范 ，在over()子句中定义，partition by和order by两部分组成。order by决定了窗口框架的范围。

窗口支持的函数

  • 连续编号
    • ROW_NUMBER()：对窗口中的行进行连续编号，从1开始。
  • 按排名编号
    • RANK()：按排序键值分配排名，相同值会得到相同的排名，并且会跳过后续的排名1,2,2,4
    • DENSE_RANK()：按排序键值分配排名，相同值会得到相同排名，但是排名是连续的1,2,2,3
    • PERCENT_RANK()：按排序键值百分比分配排名，计算逻辑是(当前行的排名-1)/(分区内总行数-1)，结果范围是[0,1]
  • NTILE(n)：将窗口内的行尽可能得平均分配成n个桶，并分配桶编号。
  • CUME_DIST()：计算规则，<=当前行排序键的行数/分区内总行数，结果范围(0,1]，对等组将会有相同的值。
  • LAG(expr, offset, default) | LEAD(expr, offset, default)：访问当行之前/之后第offset行的数据。offset默认为1，default默认为null。不受窗口规则rows和range的影响，offset是相对当前行的绝对位移。expr为想要获取值的列或表达式。
  • FIRST_VALUE(expr) | LAST_VALUE(expr)：返回窗口内expr的第一个和最后一个值，注意默认窗口规格下last_value函数的陷阱。
  • nth_value(expr,n)：计算规则：返回窗口框架内第n行的expr的值。含义同窗口内top(n)。
  • 聚合函数，几乎支持group by下的任何聚合函数

注意：

  • 在hive中over()子句中可以省略order by。当省略order by时，order by默认使用和partition by相同的字段列。
  • 在spark中必须显示指定order by，将会报错Error in query: Window function row_number() requires window to be ordered, please add ORDER BY clause。

窗口框架定义了对于当前行，其窗口的具体范围有多大。

窗口框架主要结构为：

ROWS/RANGE BETWEEN <start> AND <end>

ROWS和RANGE的主要区别是

ROWS：基于行的物理位置来划分窗口。 RANGE：基于行的排序键的值来划分窗口。

特性 ROWS RANGE
依据 物理偏移（行数） 逻辑偏移（值的范围）
计算方式 计算当前行之前/之后的具体行数 计算与当前行排序键值相差某个范围的所有行
性能 通常更快，只需移动指针 通常更慢，需要检查值是否在范围内
结果确定性 确定，顺序固定则结果固定 在存在重复值时，结果可能不直观
常用场景 计算移动平均、前后行对比（LAG/LEAD） 计算累积总和（到当前值为止）、基于值的范围分组

在标准sql中，range的行为是，包含所有行，这些行的排序键的值<=当前行的排序键的值。如果order by有重复值，这些重复值的行会被视为一个对等组(peer group)，并包含在框架内。

对等组指的是在一个已排序的数组中，所有具有相同排序键值的行的集合。在range的处理逻辑中是以值为单位，相同值被视为一个对等组表示一个不可分割的整体。

起始点 <start> 和结束点 <end> 可以是：

  • UNBOUNDED PRECEDING: 分区的第一行
  • UNBOUNDED FOLLOWING: 分区的最后一行
  • CURRENT ROW: 当前行
  • n PRECEDING: 当前行之前的第 n 行
  • n FOLLOWING: 当前行之后的第 n 行

例如：

  • ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW： 表示从分区开始到当前行。用于计算累积总和/平均值。
  • ROWS BETWEEN 3 PRECEDING AND 1 FOLLOWING： 从当前行前3行到后1行。
  • ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING： 整个分区。常用于在有序分区内求整个分区的总值。

注意： 未显示指定窗口框架时，默认为RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW。 未显示指定order by和窗口框架时，默认为ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING

2. 函数演示

创建测试表并插入测试数据

sql 复制代码 
CREATE TABLE employees (name STRING, dept STRING, salary INT, age INT);

INSERT INTO employees VALUES 
("Lisa", "Sales", 10000, 35)
,("Evan", "Sales", 32000, 38)
,("Fred", "Engineering", 21000, 28)
,("Alex", "Sales", 30000, 33)
,("Tom", "Engineering", 23000, 33)
,("Jane", "Marketing", 29000, 28)
,("Jeff", "Marketing", 35000, 38)
,("Paul", "Engineering", 29000, 23)
,("Chloe", "Engineering", 23000, 25);

2.1. 排名函数

sql 复制代码 
select *,rank() over(partition by dept order by salary) as rn from employees;
-- 在Engineering分组中，tom和chole行的排序键值相同，默认range情况下，二者将被置为一个对等组，因此编号相同
+-----------------+-----------------+-------------------+----------------+-----+
| employees.name  | employees.dept  | employees.salary  | employees.age  | rn  |
+-----------------+-----------------+-------------------+----------------+-----+
| Jane            | Marketing       | 29000             | 28             | 1   |
| Jeff            | Marketing       | 35000             | 38             | 2   |
| Fred            | Engineering     | 21000             | 28             | 1   |
| Tom             | Engineering     | 23000             | 33             | 2   |
| Chloe           | Engineering     | 23000             | 25             | 2   |
| Paul            | Engineering     | 29000             | 23             | 4   |
| Lisa            | Sales           | 10000             | 35             | 1   |
| Alex            | Sales           | 30000             | 33             | 2   |
| Evan            | Sales           | 32000             | 38             | 3   |
+-----------------+-----------------+-------------------+----------------+-----+

select *,dense_rank() over(partition by dept order by salary) as rn from employees;
+-----------------+-----------------+-------------------+----------------+-----+
| employees.name  | employees.dept  | employees.salary  | employees.age  | rn  |
+-----------------+-----------------+-------------------+----------------+-----+
| Jane            | Marketing       | 29000             | 28             | 1   |
| Jeff            | Marketing       | 35000             | 38             | 2   |
| Fred            | Engineering     | 21000             | 28             | 1   |
| Tom             | Engineering     | 23000             | 33             | 2   |
| Chloe           | Engineering     | 23000             | 25             | 2   |
| Paul            | Engineering     | 29000             | 23             | 3   |
| Lisa            | Sales           | 10000             | 35             | 1   |
| Alex            | Sales           | 30000             | 33             | 2   |
| Evan            | Sales           | 32000             | 38             | 3   |
+-----------------+-----------------+-------------------+----------------+-----+

select *,percent_rank() over(partition by dept order by salary) as rn from employees;
-- rn的结果是每行在当前分组中的百分比，注意order by中相同值结果相同
+-----------------+-----------------+-------------------+----------------+-------+
| employees.name  | employees.dept  | employees.salary  | employees.age  |  rn   |
+-----------------+-----------------+-------------------+----------------+-------+
| Jane            | Marketing       | 29000             | 28             | 0.0   | 0/1
| Jeff            | Marketing       | 35000             | 38             | 1.0   | 1/1
| Fred            | Engineering     | 21000             | 28             | 0.0   | 0/1
| Tom             | Engineering     | 23000             | 33             | 0.33  | tom和chloe并列第二，（2-1)/(4-1)
| Chloe           | Engineering     | 23000             | 25             | 0.33  |
| Paul            | Engineering     | 29000             | 23             | 1.0   | 2/2
| Lisa            | Sales           | 10000             | 35             | 0.0   | 0/2
| Alex            | Sales           | 30000             | 33             | 0.5   | 1/2
| Evan            | Sales           | 32000             | 38             | 1.0   | 2/2
+-----------------+-----------------+-------------------+----------------+-------+

select *,rank() over(partition by dept order by dept) as rn from employees;
select *,dense_rank() over(partition by dept order by dept) as rn from employees;
-- 按dept分组并按dept排名，分组内都是并列第一
+-----------------+-----------------+-------------------+----------------+-----+
| employees.name  | employees.dept  | employees.salary  | employees.age  | rn  |
+-----------------+-----------------+-------------------+----------------+-----+
| Fred            | Engineering     | 21000             | 28             | 1   |
| Tom             | Engineering     | 23000             | 33             | 1   |
| Paul            | Engineering     | 29000             | 23             | 1   |
| Chloe           | Engineering     | 23000             | 25             | 1   |
| Jane            | Marketing       | 29000             | 28             | 1   |
| Jeff            | Marketing       | 35000             | 38             | 1   |
| Lisa            | Sales           | 10000             | 35             | 1   |
| Evan            | Sales           | 32000             | 38             | 1   |
| Alex            | Sales           | 30000             | 33             | 1   |
+-----------------+-----------------+-------------------+----------------+-----+

select *,percent_rank() over(partition by dept order by dept) as rn from employees;
+-----------------+-----------------+-------------------+----------------+------+
| employees.name  | employees.dept  | employees.salary  | employees.age  |  rn  |
+-----------------+-----------------+-------------------+----------------+------+
| Fred            | Engineering     | 21000             | 28             | 0.0  |
| Tom             | Engineering     | 23000             | 33             | 0.0  |
| Paul            | Engineering     | 29000             | 23             | 0.0  |
| Chloe           | Engineering     | 23000             | 25             | 0.0  |
| Jane            | Marketing       | 29000             | 28             | 0.0  |
| Jeff            | Marketing       | 35000             | 38             | 0.0  |
| Lisa            | Sales           | 10000             | 35             | 0.0  |
| Evan            | Sales           | 32000             | 38             | 0.0  |
| Alex            | Sales           | 30000             | 33             | 0.0  |
+-----------------+-----------------+-------------------+----------------+------+

2.2. ntile(n)

作用是在分组内对数据进行分桶，然后返回桶的序号。如果数据无法按照桶个数均分，则将多余的数据放在第一个桶内。ntile不支持range窗口规格。

使用场景：例如获取每个部门薪资前三分之一的员工数据，则按照salary降序排序，然后分成3个桶，最后取第一个桶中的数据。

sql 复制代码 
select *,NTILE(2) over(partition by dept order by salary) as rn from employees;
-- 分成2桶
+-----------------+-----------------+-------------------+----------------+-----+
| employees.name  | employees.dept  | employees.salary  | employees.age  | rn  |
+-----------------+-----------------+-------------------+----------------+-----+
| Jane            | Marketing       | 29000             | 28             | 1   |
| Jeff            | Marketing       | 35000             | 38             | 2   |
| Fred            | Engineering     | 21000             | 28             | 1   |
| Tom             | Engineering     | 23000             | 33             | 1   |
| Chloe           | Engineering     | 23000             | 25             | 2   |
| Paul            | Engineering     | 29000             | 23             | 2   |
| Lisa            | Sales           | 10000             | 35             | 1   |
| Alex            | Sales           | 30000             | 33             | 1   |
| Evan            | Sales           | 32000             | 38             | 2   |
+-----------------+-----------------+-------------------+----------------+-----+
select *,NTILE(3) over(partition by dept order by salary) as rn from employees;
-- 分成3桶
+-----------------+-----------------+-------------------+----------------+-----+
| employees.name  | employees.dept  | employees.salary  | employees.age  | rn  |
+-----------------+-----------------+-------------------+----------------+-----+
| Jane            | Marketing       | 29000             | 28             | 1   |
| Jeff            | Marketing       | 35000             | 38             | 2   |
| Fred            | Engineering     | 21000             | 28             | 1   |
| Tom             | Engineering     | 23000             | 33             | 1   |
| Chloe           | Engineering     | 23000             | 25             | 2   |
| Paul            | Engineering     | 29000             | 23             | 3   |
| Lisa            | Sales           | 10000             | 35             | 1   |
| Alex            | Sales           | 30000             | 33             | 2   |
| Evan            | Sales           | 32000             | 38             | 3   |
+-----------------+-----------------+-------------------+----------------+-----+

2.3. cume_dist()

sql 复制代码 
select *,cume_dist() over(partition by dept order by salary) from employees;
-- enginnering分组中，tom和chloe中salary相同，二者的分位数结果相同
+-----------------+-----------------+-------------------+----------------+---------------------+
| employees.name  | employees.dept  | employees.salary  | employees.age  | cume_dist_window_0  |
+-----------------+-----------------+-------------------+----------------+---------------------+
| Jane            | Marketing       | 29000             | 28             | 0.5                 | 1/2
| Jeff            | Marketing       | 35000             | 38             | 1.0                 | 2/2
| Fred            | Engineering     | 21000             | 28             | 0.25                | 1/4
| Tom             | Engineering     | 23000             | 33             | 0.75                | 3/4
| Chloe           | Engineering     | 23000             | 25             | 0.75                | 3/4
| Paul            | Engineering     | 29000             | 23             | 1.0                 | 4/4
| Lisa            | Sales           | 10000             | 35             | 0.3333333333333333  | 1/3
| Alex            | Sales           | 30000             | 33             | 0.6666666666666666  | 2/3
| Evan            | Sales           | 32000             | 38             | 1.0                 | 3/3
+-----------------+-----------------+-------------------+----------------+---------------------+

select *,cume_dist() over(partition by dept order by age) from employees;
-- 按照age排序，tom和chloe中age不同，因此分位数不同
+-----------------+-----------------+-------------------+----------------+---------------------+
| employees.name  | employees.dept  | employees.salary  | employees.age  | cume_dist_window_0  |
+-----------------+-----------------+-------------------+----------------+---------------------+
| Jane            | Marketing       | 29000             | 28             | 0.5                 |
| Jeff            | Marketing       | 35000             | 38             | 1.0                 |
| Paul            | Engineering     | 29000             | 23             | 0.25                |
| Chloe           | Engineering     | 23000             | 25             | 0.5                 |
| Fred            | Engineering     | 21000             | 28             | 0.75                |
| Tom             | Engineering     | 23000             | 33             | 1.0                 |
| Alex            | Sales           | 30000             | 33             | 0.3333333333333333  |
| Lisa            | Sales           | 10000             | 35             | 0.6666666666666666  |
| Evan            | Sales           | 32000             | 38             | 1.0                 |
+-----------------+-----------------+-------------------+----------------+---------------------+

2.4. LAG/LEAD(expr,offset,default_value)

lag 表示落后的含义，在使用场景中是小于的含义。 lead 表示领先的含义，在使用场景中是大于的含义。

可以实现不自连接的前提下，按照order by结果得到当前行指定列 前/后移动num行 的列值。

函数参数 LAG/LEAD(expr,offset,default_value)：expr表示指定的列名；offset表示指定的位移行数，默认为1；default_value表示末尾或开头返回的值，默认值null。

sql 复制代码 
SELECT *,
    LAG(salary) OVER (PARTITION BY dept ORDER BY salary) AS lag,
    LEAD(salary) OVER (PARTITION BY dept ORDER BY salary) AS lead
    FROM employees;
-- 从结果看，按照order by排序结果，lag取的值是当前行前1行的值，lead取的值是当前行后1行的值，对于第1行或者最后1行，取值默认为null
+-----------------+-----------------+-------------------+----------------+--------+--------+
| employees.name  | employees.dept  | employees.salary  | employees.age  |  lag   |  lead  |
+-----------------+-----------------+-------------------+----------------+--------+--------+
| Jane            | Marketing       | 29000             | 28             | NULL   | 35000  |
| Jeff            | Marketing       | 35000             | 38             | 29000  | NULL   |
| Fred            | Engineering     | 21000             | 28             | NULL   | 23000  |
| Tom             | Engineering     | 23000             | 33             | 21000  | 23000  |
| Chloe           | Engineering     | 23000             | 25             | 23000  | 29000  |
| Paul            | Engineering     | 29000             | 23             | 23000  | NULL   |
| Lisa            | Sales           | 10000             | 35             | NULL   | 30000  |
| Alex            | Sales           | 30000             | 33             | 10000  | 32000  |
| Evan            | Sales           | 32000             | 38             | 30000  | NULL   |
+-----------------+-----------------+-------------------+----------------+--------+--------+

SELECT *,
    LAG(salary,2,0) OVER (PARTITION BY dept ORDER BY salary) AS lag,
    LEAD(salary, 1,0) OVER (PARTITION BY dept ORDER BY salary) AS lead
    FROM employees;
+-----------------+-----------------+-------------------+----------------+--------+--------+
| employees.name  | employees.dept  | employees.salary  | employees.age  |  lag   |  lead  |
+-----------------+-----------------+-------------------+----------------+--------+--------+
| Jane            | Marketing       | 29000             | 28             | 0      | 35000  |
| Jeff            | Marketing       | 35000             | 38             | 0      | 0      |
| Fred            | Engineering     | 21000             | 28             | 0      | 23000  |
| Tom             | Engineering     | 23000             | 33             | 0      | 23000  |
| Chloe           | Engineering     | 23000             | 25             | 21000  | 29000  |
| Paul            | Engineering     | 29000             | 23             | 23000  | 0      |
| Lisa            | Sales           | 10000             | 35             | 0      | 30000  |
| Alex            | Sales           | 30000             | 33             | 0      | 32000  |
| Evan            | Sales           | 32000             | 38             | 10000  | 0      |
+-----------------+-----------------+-------------------+----------------+--------+--------+

-- 排序结果和取值列不同的情况
SELECT *,
    LAG(salary) OVER (PARTITION BY dept ORDER BY age) AS lag,
    LEAD(salary) OVER (PARTITION BY dept ORDER BY age) AS lead
    FROM employees;
+-----------------+-----------------+-------------------+----------------+--------+--------+
| employees.name  | employees.dept  | employees.salary  | employees.age  |  lag   |  lead  |
+-----------------+-----------------+-------------------+----------------+--------+--------+
| Jane            | Marketing       | 29000             | 28             | NULL   | 35000  |
| Jeff            | Marketing       | 35000             | 38             | 29000  | NULL   |
| Paul            | Engineering     | 29000             | 23             | NULL   | 23000  |
| Chloe           | Engineering     | 23000             | 25             | 29000  | 21000  |
| Fred            | Engineering     | 21000             | 28             | 23000  | 23000  |
| Tom             | Engineering     | 23000             | 33             | 21000  | NULL   |
| Alex            | Sales           | 30000             | 33             | NULL   | 10000  |
| Lisa            | Sales           | 10000             | 35             | 30000  | 32000  |
| Evan            | Sales           | 32000             | 38             | 10000  | NULL   |
+-----------------+-----------------+-------------------+----------------+--------+--------+

2.5. first_value/last_value(expr)

在分组中按照order by的结果，获取指定列的第一个或最后一个值。

注意，默认情况下last_value取的是第一行截止到当前行的最后一个值(当前行的值)，并不是整个分区中排序后的最后一个值。

sql 复制代码 
SELECT *,
    first_value(salary) OVER (PARTITION BY dept ORDER BY age) AS first,
    last_value(salary) OVER (PARTITION BY dept ORDER BY age) AS last
    FROM employees;
-- 注意last的结果
+-----------------+-----------------+-------------------+----------------+--------+--------+
| employees.name  | employees.dept  | employees.salary  | employees.age  | first  |  last  |
+-----------------+-----------------+-------------------+----------------+--------+--------+
| Jane            | Marketing       | 29000             | 28             | 29000  | 29000  |
| Jeff            | Marketing       | 35000             | 38             | 29000  | 35000  |
| Paul            | Engineering     | 29000             | 23             | 29000  | 29000  |
| Chloe           | Engineering     | 23000             | 25             | 29000  | 23000  |
| Fred            | Engineering     | 21000             | 28             | 29000  | 21000  |
| Tom             | Engineering     | 23000             | 33             | 29000  | 23000  |
| Alex            | Sales           | 30000             | 33             | 30000  | 30000  |
| Lisa            | Sales           | 10000             | 35             | 30000  | 10000  |
| Evan            | Sales           | 32000             | 38             | 30000  | 32000  |
+-----------------+-----------------+-------------------+----------------+--------+--------+

SELECT *,
    first_value(salary) OVER (PARTITION BY dept ORDER BY salary) AS first,
    last_value(salary) OVER (PARTITION BY dept ORDER BY salary) AS last
    FROM employees;
+-----------------+-----------------+-------------------+----------------+--------+--------+
| employees.name  | employees.dept  | employees.salary  | employees.age  | first  |  last  |
+-----------------+-----------------+-------------------+----------------+--------+--------+
| Jane            | Marketing       | 29000             | 28             | 29000  | 29000  |
| Jeff            | Marketing       | 35000             | 38             | 29000  | 35000  |
| Fred            | Engineering     | 21000             | 28             | 21000  | 21000  |
| Tom             | Engineering     | 23000             | 33             | 21000  | 23000  |
| Chloe           | Engineering     | 23000             | 25             | 21000  | 23000  |
| Paul            | Engineering     | 29000             | 23             | 21000  | 29000  |
| Lisa            | Sales           | 10000             | 35             | 10000  | 10000  |
| Alex            | Sales           | 30000             | 33             | 10000  | 30000  |
| Evan            | Sales           | 32000             | 38             | 10000  | 32000  |
+-----------------+-----------------+-------------------+----------------+--------+--------+

陷阱： last_value在默认的窗口规格下RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW，每行的为当前行，因此last结果为当前行的salary值。

2.6. nth_value(expr,n)

nth表示第几个的含义

作用，在分组中返回order by结果中指定列的第N行值。

注意，hive中无此函数

sql 复制代码 
SELECT *,nth_value(salary,2) OVER (PARTITION BY dept ORDER BY salary) AS nth FROM employees;
相关推荐
鸿乃江边鸟8 小时前
Spark Datafusion Comet 向量化Rust Native--CometShuffleExchangeExec怎么控制读写
大数据·rust·spark·native
伟大的大威1 天前
NVIDIA DGX Spark (ARM64/Blackwell) Kubernetes 集群 + GPU Operator 完整部署指南
大数据·spark·kubernetes
小邓睡不饱耶1 天前
深度实战：Spark GraphX构建用户信任网络，精准锁定高价值目标用户（含完整案例）
大数据·spark·php
B站计算机毕业设计超人1 天前
计算机毕业设计hadoop+spark+hive共享单车预测系统 共享单车数据可视化分析 大数据毕业设计(源码+LW文档+PPT+讲解)
大数据·hadoop·python·深度学习·spark·毕业设计·课程设计
B站计算机毕业设计超人1 天前
计算机毕业设计Python+Spark+Hadoop+Hive微博舆情分析 微博情感分析可视化 大数据毕业设计(源码+LW文档+PPT+讲解)
大数据·hadoop·爬虫·python·spark·cnn·课程设计
yumgpkpm1 天前
华为昇腾300T A2训练、微调Qwen过程，带保姆式命令，麒麟操作系统+鲲鹏CPU
hive·hadoop·华为·flink·spark·kafka·hbase
TTBIGDATA1 天前
【Hue】Ambari开启 Kerberos 后，Hue 使用 Spark SQL出现凭证不统一问题处理
大数据·sql·spark·ambari·kerberos·hue·bigtop
鸿乃江边鸟2 天前
Spark Datafusion Comet 向量化Rust Native--Native算子(CometNativeExec)怎么串联执行
大数据·rust·spark·native
Light602 天前
数智孪生，金流·物流全透视：构建某银行制造业贷后风控新范式—— 基于领码 SPARK 融合平台的技术解决方案
大数据·spark·数字孪生·实时监控·物联网金融·供应链风控·ai决策
小邓睡不饱耶2 天前
基于Spark GraphX构建用户信任网络：精准定位高价值目标用户
大数据·spark·php
热门推荐
01GitHub 镜像站点02Claude Code + GLM4.7 避坑指南：解决 Unable to connect to Anthropic services03使用 1panel面板 部署 php网站04Linux下V2Ray安装配置指南05OpenClaw Chrome扩展使用教程 - 浏览器中继控制06Vue-skills的中文文档07UV安装并设置国内源08让 Trae IDE 智能体 “读懂”文档 Excel+PDF+DOCX ：mcp-documents-reader 工具使用指南09Claude Code Skills 实用使用手册10openclaw配置教程（linux+局域网ollama）