Hive/Spark窗口函数

SunnyZ-L2024-07-13 0:34

窗口函数

hive文档链接
spark文档链接

1. OVER支持的函数

  • 自然序编号
    Syntax: ROW_NUMBER
  • 按等级编号
    Syntax: RANK | DENSE_RANK | PERCENT_RANK
  • 分组内分桶，并返回对应桶的序号
    Syntax: NTILE(n)
  • Analytic Functions（分析函数）
    Syntax: CUME_DIST | LAG | LEAD | NTH_VALUE | FIRST_VALUE | LAST_VALUE
  • Aggregate Functions（聚合函数）
    Syntax: MAX | MIN | COUNT | SUM | AVG | ...

1.1. 准备工作

创建测试表并插入测试数据

sql 复制代码 
CREATE TABLE employees (name STRING, dept STRING, salary INT, age INT);

INSERT INTO employees VALUES 
("Lisa", "Sales", 10000, 35)
,("Evan", "Sales", 32000, 38)
,("Fred", "Engineering", 21000, 28)
,("Alex", "Sales", 30000, 33)
,("Tom", "Engineering", 23000, 33)
,("Jane", "Marketing", 29000, 28)
,("Jeff", "Marketing", 35000, 38)
,("Paul", "Engineering", 29000, 23)
,("Chloe", "Engineering", 23000, 25);

1.2. row_number()

row_number() over() row_number可能是窗口函数中使用最频繁的函数，作用是在分组内按自然序进行编号，结果值为：1 2 3 4 5

sql 复制代码 
select *,row_number() over(partition by dept order by salary) as rn from employees;
+-----------------+-----------------+-------------------+----------------+-----+
| employees.name  | employees.dept  | employees.salary  | employees.age  | rn  |
+-----------------+-----------------+-------------------+----------------+-----+
| Jane            | Marketing       | 29000             | 28             | 1   |
| Jeff            | Marketing       | 35000             | 38             | 2   |
| Fred            | Engineering     | 21000             | 28             | 1   |
| Tom             | Engineering     | 23000             | 33             | 2   |
| Chloe           | Engineering     | 23000             | 25             | 3   |
| Paul            | Engineering     | 29000             | 23             | 4   |
| Lisa            | Sales           | 10000             | 35             | 1   |
| Alex            | Sales           | 30000             | 33             | 2   |
| Evan            | Sales           | 32000             | 38             | 3   |
+-----------------+-----------------+-------------------+----------------+-----+

1.3. rank函数

rank有等级的含义，函数作用是在分组内按照order by的结果得到有等级编号。
rank() over () 并列有间隔，结果如：1 2 2 4 5
dense_rank() over() dense有密集的含义，并列无间隔，结果如：1 2 2 3 4
percent_rank() over() 百分比的rank，含义是(当前行-1)/(当前组总行数-1)，当前行从1开始

注意，如果order by的结果相同，则rank得到的结果都相同，在这里的语义是排序结果相同，因此等级编号也相同。

sql 复制代码 
select *,rank() over(partition by dept order by salary) as rn from employees;
-- 执行结果中，engineering中salary相同的编号相同，paul的值为4
+-----------------+-----------------+-------------------+----------------+-----+
| employees.name  | employees.dept  | employees.salary  | employees.age  | rn  |
+-----------------+-----------------+-------------------+----------------+-----+
| Jane            | Marketing       | 29000             | 28             | 1   |
| Jeff            | Marketing       | 35000             | 38             | 2   |
| Fred            | Engineering     | 21000             | 28             | 1   |
| Tom             | Engineering     | 23000             | 33             | 2   |
| Chloe           | Engineering     | 23000             | 25             | 2   |
| Paul            | Engineering     | 29000             | 23             | 4   |
| Lisa            | Sales           | 10000             | 35             | 1   |
| Alex            | Sales           | 30000             | 33             | 2   |
| Evan            | Sales           | 32000             | 38             | 3   |
+-----------------+-----------------+-------------------+----------------+-----+

select *,dense_rank() over(partition by dept order by salary) as rn from employees;
-- 执行结果中，engineering中salary相同的编号相同，paul的值为3
+-----------------+-----------------+-------------------+----------------+-----+
| employees.name  | employees.dept  | employees.salary  | employees.age  | rn  |
+-----------------+-----------------+-------------------+----------------+-----+
| Jane            | Marketing       | 29000             | 28             | 1   |
| Jeff            | Marketing       | 35000             | 38             | 2   |
| Fred            | Engineering     | 21000             | 28             | 1   |
| Tom             | Engineering     | 23000             | 33             | 2   |
| Chloe           | Engineering     | 23000             | 25             | 2   |
| Paul            | Engineering     | 29000             | 23             | 3   |
| Lisa            | Sales           | 10000             | 35             | 1   |
| Alex            | Sales           | 30000             | 33             | 2   |
| Evan            | Sales           | 32000             | 38             | 3   |
+-----------------+-----------------+-------------------+----------------+-----+

select *,percent_rank() over(partition by dept order by salary) as rn from employees;
-- rn的结果是每行在当前分组中的百分比，注意order by中相同值结果相同
+-----------------+-----------------+-------------------+----------------+---------------------+
| employees.name  | employees.dept  | employees.salary  | employees.age  |         rn          |
+-----------------+-----------------+-------------------+----------------+---------------------+
| Jane            | Marketing       | 29000             | 28             | 0.0                 |
| Jeff            | Marketing       | 35000             | 38             | 1.0                 |
| Fred            | Engineering     | 21000             | 28             | 0.0                 |
| Tom             | Engineering     | 23000             | 33             | 0.3333333333333333  |
| Chloe           | Engineering     | 23000             | 25             | 0.3333333333333333  |
| Paul            | Engineering     | 29000             | 23             | 1.0                 |
| Lisa            | Sales           | 10000             | 35             | 0.0                 |
| Alex            | Sales           | 30000             | 33             | 0.5                 |
| Evan            | Sales           | 32000             | 38             | 1.0                 |
+-----------------+-----------------+-------------------+----------------+---------------------+

-- order by结果相同的情况，order by dept
select *,rank() over(partition by dept order by dept) as rn from employees;
select *,dense_rank() over(partition by dept order by dept) as rn from employees;
-- 二者结果相同
+-----------------+-----------------+-------------------+----------------+-----+
| employees.name  | employees.dept  | employees.salary  | employees.age  | rn  |
+-----------------+-----------------+-------------------+----------------+-----+
| Fred            | Engineering     | 21000             | 28             | 1   |
| Tom             | Engineering     | 23000             | 33             | 1   |
| Paul            | Engineering     | 29000             | 23             | 1   |
| Chloe           | Engineering     | 23000             | 25             | 1   |
| Jane            | Marketing       | 29000             | 28             | 1   |
| Jeff            | Marketing       | 35000             | 38             | 1   |
| Lisa            | Sales           | 10000             | 35             | 1   |
| Evan            | Sales           | 32000             | 38             | 1   |
| Alex            | Sales           | 30000             | 33             | 1   |
+-----------------+-----------------+-------------------+----------------+-----+

select *,percent_rank() over(partition by dept order by dept) as rn from employees;
+-----------------+-----------------+-------------------+----------------+------+
| employees.name  | employees.dept  | employees.salary  | employees.age  |  rn  |
+-----------------+-----------------+-------------------+----------------+------+
| Fred            | Engineering     | 21000             | 28             | 0.0  |
| Tom             | Engineering     | 23000             | 33             | 0.0  |
| Paul            | Engineering     | 29000             | 23             | 0.0  |
| Chloe           | Engineering     | 23000             | 25             | 0.0  |
| Jane            | Marketing       | 29000             | 28             | 0.0  |
| Jeff            | Marketing       | 35000             | 38             | 0.0  |
| Lisa            | Sales           | 10000             | 35             | 0.0  |
| Evan            | Sales           | 32000             | 38             | 0.0  |
| Alex            | Sales           | 30000             | 33             | 0.0  |
+-----------------+-----------------+-------------------+----------------+------+

1.4. ntile(n)

ntile(n) over tile有瓷砖、瓦片的含义，作用是在分组内对数据进行分桶，然后返回桶的序号。

按照order by结果将数据平均的放入到分好的桶中，如果数据无法按照桶个数均分，则将多余的数据放在第一个桶内。

使用场景：例如获取每个部门薪资前三分之一的员工数据，则按照salary降序排序，然后分成3个桶，最后取第一个桶中的数据。

sql 复制代码 
select *,NTILE(2) over(partition by dept order by salary) as rn from employees;
-- ntile(2)表示分成2桶，
+-----------------+-----------------+-------------------+----------------+-----+
| employees.name  | employees.dept  | employees.salary  | employees.age  | rn  |
+-----------------+-----------------+-------------------+----------------+-----+
| Jane            | Marketing       | 29000             | 28             | 1   |
| Jeff            | Marketing       | 35000             | 38             | 2   |
| Fred            | Engineering     | 21000             | 28             | 1   |
| Tom             | Engineering     | 23000             | 33             | 1   |
| Chloe           | Engineering     | 23000             | 25             | 2   |
| Paul            | Engineering     | 29000             | 23             | 2   |
| Lisa            | Sales           | 10000             | 35             | 1   |
| Alex            | Sales           | 30000             | 33             | 1   |
| Evan            | Sales           | 32000             | 38             | 2   |
+-----------------+-----------------+-------------------+----------------+-----+
select *,NTILE(3) over(partition by dept order by salary) as rn from employees;
-- 分成3桶
+-----------------+-----------------+-------------------+----------------+-----+
| employees.name  | employees.dept  | employees.salary  | employees.age  | rn  |
+-----------------+-----------------+-------------------+----------------+-----+
| Jane            | Marketing       | 29000             | 28             | 1   |
| Jeff            | Marketing       | 35000             | 38             | 2   |
| Fred            | Engineering     | 21000             | 28             | 1   |
| Tom             | Engineering     | 23000             | 33             | 1   |
| Chloe           | Engineering     | 23000             | 25             | 2   |
| Paul            | Engineering     | 29000             | 23             | 3   |
| Lisa            | Sales           | 10000             | 35             | 1   |
| Alex            | Sales           | 30000             | 33             | 2   |
| Evan            | Sales           | 32000             | 38             | 3   |
+-----------------+-----------------+-------------------+----------------+-----+

1.5. cume_dist()

cume表示累计，dist表示距离

用于求累计分布，即分组中按照order by结果的分位数

sql 复制代码 
select *,cume_dist() over(partition by dept order by salary) from employees;
-- enginnering分组中，tom和chloe中salary相同，二者的分位数结果相同
+-----------------+-----------------+-------------------+----------------+---------------------+
| employees.name  | employees.dept  | employees.salary  | employees.age  | cume_dist_window_0  |
+-----------------+-----------------+-------------------+----------------+---------------------+
| Jane            | Marketing       | 29000             | 28             | 0.5                 |
| Jeff            | Marketing       | 35000             | 38             | 1.0                 |
| Fred            | Engineering     | 21000             | 28             | 0.25                |
| Tom             | Engineering     | 23000             | 33             | 0.75                |
| Chloe           | Engineering     | 23000             | 25             | 0.75                |
| Paul            | Engineering     | 29000             | 23             | 1.0                 |
| Lisa            | Sales           | 10000             | 35             | 0.3333333333333333  |
| Alex            | Sales           | 30000             | 33             | 0.6666666666666666  |
| Evan            | Sales           | 32000             | 38             | 1.0                 |
+-----------------+-----------------+-------------------+----------------+---------------------+

select *,cume_dist() over(partition by dept order by age) from employees;
-- 按照age排序，tom和chloe中age不同，因此分位数不同
+-----------------+-----------------+-------------------+----------------+---------------------+
| employees.name  | employees.dept  | employees.salary  | employees.age  | cume_dist_window_0  |
+-----------------+-----------------+-------------------+----------------+---------------------+
| Jane            | Marketing       | 29000             | 28             | 0.5                 |
| Jeff            | Marketing       | 35000             | 38             | 1.0                 |
| Paul            | Engineering     | 29000             | 23             | 0.25                |
| Chloe           | Engineering     | 23000             | 25             | 0.5                 |
| Fred            | Engineering     | 21000             | 28             | 0.75                |
| Tom             | Engineering     | 23000             | 33             | 1.0                 |
| Alex            | Sales           | 30000             | 33             | 0.3333333333333333  |
| Lisa            | Sales           | 10000             | 35             | 0.6666666666666666  |
| Evan            | Sales           | 32000             | 38             | 1.0                 |
+-----------------+-----------------+-------------------+----------------+---------------------+

1.6. lag和lead

lag 表示落后的含义，在使用场景中是小于的含义。

lead 表示领先的含义，在使用场景中是大于的含义。

可以实现不自连接的前提下，按照order by结果得到当前行指定列 前/后移动num行 的列值。

函数参数
LAG/LEAD(col,num,default_value)：col表示指定的列名；num表示指定的位移行数，默认为1；default_value表示末尾或开头返回的值，默认值null。

sql 复制代码 
SELECT *,
    LAG(salary) OVER (PARTITION BY dept ORDER BY salary) AS lag,
    LEAD(salary) OVER (PARTITION BY dept ORDER BY salary) AS lead
    FROM employees;
-- 从结果看，按照order by排序结果，lag取的值是当前行前1行的值，lead取的值是当前行后1行的值，对于第1行或者最后1行，取值默认为null
+-----------------+-----------------+-------------------+----------------+--------+--------+
| employees.name  | employees.dept  | employees.salary  | employees.age  |  lag   |  lead  |
+-----------------+-----------------+-------------------+----------------+--------+--------+
| Jane            | Marketing       | 29000             | 28             | NULL   | 35000  |
| Jeff            | Marketing       | 35000             | 38             | 29000  | NULL   |
| Fred            | Engineering     | 21000             | 28             | NULL   | 23000  |
| Tom             | Engineering     | 23000             | 33             | 21000  | 23000  |
| Chloe           | Engineering     | 23000             | 25             | 23000  | 29000  |
| Paul            | Engineering     | 29000             | 23             | 23000  | NULL   |
| Lisa            | Sales           | 10000             | 35             | NULL   | 30000  |
| Alex            | Sales           | 30000             | 33             | 10000  | 32000  |
| Evan            | Sales           | 32000             | 38             | 30000  | NULL   |
+-----------------+-----------------+-------------------+----------------+--------+--------+

SELECT *,
    LAG(salary,2,0) OVER (PARTITION BY dept ORDER BY salary) AS lag,
    LEAD(salary, 1, 0) OVER (PARTITION BY dept ORDER BY salary) AS lead
    FROM employees;
+-----------------+-----------------+-------------------+----------------+--------+--------+
| employees.name  | employees.dept  | employees.salary  | employees.age  |  lag   |  lead  |
+-----------------+-----------------+-------------------+----------------+--------+--------+
| Jane            | Marketing       | 29000             | 28             | 0      | 35000  |
| Jeff            | Marketing       | 35000             | 38             | 0      | 0      |
| Fred            | Engineering     | 21000             | 28             | 0      | 23000  |
| Tom             | Engineering     | 23000             | 33             | 0      | 23000  |
| Chloe           | Engineering     | 23000             | 25             | 21000  | 29000  |
| Paul            | Engineering     | 29000             | 23             | 23000  | 0      |
| Lisa            | Sales           | 10000             | 35             | 0      | 30000  |
| Alex            | Sales           | 30000             | 33             | 0      | 32000  |
| Evan            | Sales           | 32000             | 38             | 10000  | 0      |
+-----------------+-----------------+-------------------+----------------+--------+--------+

-- 排序结果和取值列不同的情况
SELECT *,
    LAG(salary) OVER (PARTITION BY dept ORDER BY age) AS lag,
    LEAD(salary) OVER (PARTITION BY dept ORDER BY age) AS lead
    FROM employees;
+-----------------+-----------------+-------------------+----------------+--------+--------+
| employees.name  | employees.dept  | employees.salary  | employees.age  |  lag   |  lead  |
+-----------------+-----------------+-------------------+----------------+--------+--------+
| Jane            | Marketing       | 29000             | 28             | NULL   | 35000  |
| Jeff            | Marketing       | 35000             | 38             | 29000  | NULL   |
| Paul            | Engineering     | 29000             | 23             | NULL   | 23000  |
| Chloe           | Engineering     | 23000             | 25             | 29000  | 21000  |
| Fred            | Engineering     | 21000             | 28             | 23000  | 23000  |
| Tom             | Engineering     | 23000             | 33             | 21000  | NULL   |
| Alex            | Sales           | 30000             | 33             | NULL   | 10000  |
| Lisa            | Sales           | 10000             | 35             | 30000  | 32000  |
| Evan            | Sales           | 32000             | 38             | 10000  | NULL   |
+-----------------+-----------------+-------------------+----------------+--------+--------+

1.7. first_value和last_value

在分组中按照order by的结果，获取指定列的第一个或最后一个值。

注意，默认情况下last_value取的是第一行截止到当前行的最后一个值(当前行的值)，并不是整个分区中排序后的最后一个值。

sql 复制代码 
SELECT *,
    first_value(salary) OVER (PARTITION BY dept ORDER BY age) AS first,
    last_value(salary) OVER (PARTITION BY dept ORDER BY age) AS last
    FROM employees;
-- 注意last的结果
+-----------------+-----------------+-------------------+----------------+--------+--------+
| employees.name  | employees.dept  | employees.salary  | employees.age  | first  |  last  |
+-----------------+-----------------+-------------------+----------------+--------+--------+
| Jane            | Marketing       | 29000             | 28             | 29000  | 29000  |
| Jeff            | Marketing       | 35000             | 38             | 29000  | 35000  |
| Paul            | Engineering     | 29000             | 23             | 29000  | 29000  |
| Chloe           | Engineering     | 23000             | 25             | 29000  | 23000  |
| Fred            | Engineering     | 21000             | 28             | 29000  | 21000  |
| Tom             | Engineering     | 23000             | 33             | 29000  | 23000  |
| Alex            | Sales           | 30000             | 33             | 30000  | 30000  |
| Lisa            | Sales           | 10000             | 35             | 30000  | 10000  |
| Evan            | Sales           | 32000             | 38             | 30000  | 32000  |
+-----------------+-----------------+-------------------+----------------+--------+--------+

SELECT *,
    first_value(salary) OVER (PARTITION BY dept ORDER BY salary) AS first,
    last_value(salary) OVER (PARTITION BY dept ORDER BY salary) AS last
    FROM employees;
+-----------------+-----------------+-------------------+----------------+--------+--------+
| employees.name  | employees.dept  | employees.salary  | employees.age  | first  |  last  |
+-----------------+-----------------+-------------------+----------------+--------+--------+
| Jane            | Marketing       | 29000             | 28             | 29000  | 29000  |
| Jeff            | Marketing       | 35000             | 38             | 29000  | 35000  |
| Fred            | Engineering     | 21000             | 28             | 21000  | 21000  |
| Tom             | Engineering     | 23000             | 33             | 21000  | 23000  |
| Chloe           | Engineering     | 23000             | 25             | 21000  | 23000  |
| Paul            | Engineering     | 29000             | 23             | 21000  | 29000  |
| Lisa            | Sales           | 10000             | 35             | 10000  | 10000  |
| Alex            | Sales           | 30000             | 33             | 10000  | 30000  |
| Evan            | Sales           | 32000             | 38             | 10000  | 32000  |
+-----------------+-----------------+-------------------+----------------+--------+--------+

1.8. nth_value

nth表示第几个的含义

作用，在分组中返回order by结果中指定列的第N行值。

注意，hive中无此函数

sql 复制代码 
SELECT *,nth_value(salary,2) OVER (PARTITION BY dept ORDER BY salary) AS nth FROM employees;

2. OVER从句

  • 在hive中over语句支持仅有partition by语句，没有order by语句。当没有order by时，order by默认使用partition by指定的字段序列，
  • 在spark中如果未指定order by，将会报错Error in query: Window function row_number() requires window to be ordered, please add ORDER BY clause
sql 复制代码 
select *, row_number() over(partition by name) as rn from employees;

-- 执行结果如下，从结果中得知按照name的字典序排序
+-----------------+-----------------+-------------------+----------------+-----+
| employees.name  | employees.dept  | employees.salary  | employees.age  | rn  |
+-----------------+-----------------+-------------------+----------------+-----+
| Alex            | Sales           | 30000             | 33             | 1   |
| Evan            | Sales           | 32000             | 38             | 1   |
| Fred            | Engineering     | 21000             | 28             | 1   |
| Lisa            | Sales           | 10000             | 35             | 1   |
| Tom             | Engineering     | 23000             | 33             | 1   |
| Chloe           | Engineering     | 23000             | 25             | 1   |
| Jane            | Marketing       | 29000             | 28             | 1   |
| Jeff            | Marketing       | 35000             | 38             | 1   |
| Paul            | Engineering     | 29000             | 23             | 1   |
+-----------------+-----------------+-------------------+----------------+-----+

2.1. window specification

在OVER语句中可以带有一个window specification，支持以下格式：

sql 复制代码 
格式：
(ROWS | RANGE) BETWEEN xxx AND xxx
具体支持以下3种格式
(ROWS | RANGE) BETWEEN (UNBOUNDED | [num]) PRECEDING AND ([num] PRECEDING | CURRENT ROW | (UNBOUNDED | [num]) FOLLOWING)
(ROWS | RANGE) BETWEEN CURRENT ROW AND (CURRENT ROW | (UNBOUNDED | [num]) FOLLOWING)
(ROWS | RANGE) BETWEEN [num] FOLLOWING AND (UNBOUNDED | [num]) FOLLOWING
  • ROWS 表示在同分区中，按照order by结果按行进行逐行处理
  • RANGE 表示在同分区中按照order by结果的范围处理，如1,2,3,3,4,4,5排序中，3,3和4,4的排序结果相同，则3,3和4,4将分别当成整体处理，每行对应开窗函数的结果相同。
  • UNBOUNDED PRECEDING 表示未绑定当前行之前的行。整个分区中，从第一行开始处理
  • num PRECEDING 表示限制当前行的前num行。如在sum时，每行sum的结果是从前num行到当前行的累加值
  • CURRENT ROW 表示处理过程中的当前行（处理数据过程中的游标指针）
  • UNBOUNDED FOLLOWING 表示未绑定当前行之后的行。整个分区，处理到最后一行
  • num FOLLOWING 表示限制当前行的后num行。如在sum时，每行值的sum的结果将处理到后num行。

注意：在hive中函数不支持使用window specification
rank,、dense_rank、percent_rank()
cume_dist()、ntile
lead、lag

2.1.1. 显式order by下的默认值

当指定了order by语句而未指定window specification语句时，默认的window specification语句是RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW。

即按照order by结果的range处理，并且从分区中的第一行处理到当前行。

sql 复制代码 
select *,sum(salary) over(partition by dept order by salary) as sum_salary from employees;
select *,sum(salary) over(partition by dept order by salary range between unbounded preceding and current row) as sum_salary from employees;

-- 二者的执行结果相同
+-----------------+-----------------+-------------------+----------------+-------------+
| employees.name  | employees.dept  | employees.salary  | employees.age  | sum_salary  |
+-----------------+-----------------+-------------------+----------------+-------------+
| Jane            | Marketing       | 29000             | 28             | 29000       |
| Jeff            | Marketing       | 35000             | 38             | 64000       |
| Fred            | Engineering     | 21000             | 28             | 21000       |
| Tom             | Engineering     | 23000             | 33             | 67000       |
| Chloe           | Engineering     | 23000             | 25             | 67000       |
| Paul            | Engineering     | 29000             | 23             | 96000       |
| Lisa            | Sales           | 10000             | 35             | 10000       |
| Alex            | Sales           | 30000             | 33             | 40000       |
| Evan            | Sales           | 32000             | 38             | 72000       |
+-----------------+-----------------+-------------------+----------------+-------------+

注意：Engineering部门的tom和chloe员工的sum_salary值相同。这是因为在Engineering分区下，二者order by salary结果相同。在range策略下，tom和chloe将当成整体处理，即46000，因此tom和chloe的累计值都为67000（21000+460000），而不是44000和67000。

在SQL中显示指定如下语句，实现分区中的按行累加 效果，即
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW

Engineering部门的tom和chloe员工的sum_salary结果将分别为44000和67000。

sql 复制代码 
select *,sum(salary) over(partition by dept order by salary rows between unbounded preceding and current row) as sum_salary from employees;

-- 结果如下
+-----------------+-----------------+-------------------+----------------+-------------+
| employees.name  | employees.dept  | employees.salary  | employees.age  | sum_salary  |
+-----------------+-----------------+-------------------+----------------+-------------+
| Jane            | Marketing       | 29000             | 28             | 29000       |
| Jeff            | Marketing       | 35000             | 38             | 64000       |
| Fred            | Engineering     | 21000             | 28             | 21000       |
| Tom             | Engineering     | 23000             | 33             | 44000       |
| Chloe           | Engineering     | 23000             | 25             | 67000       |
| Paul            | Engineering     | 29000             | 23             | 96000       |
| Lisa            | Sales           | 10000             | 35             | 10000       |
| Alex            | Sales           | 30000             | 33             | 40000       |
| Evan            | Sales           | 32000             | 38             | 72000       |
+-----------------+-----------------+-------------------+----------------+-------------+

在SQL中显示指定如下语句，实现相同分组内进行全部值求和 效果。
RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING

sql 复制代码 
select *,sum(salary) over(partition by dept order by salary range between unbounded preceding and unbounded following) as sum_salary from employees;

-- 结果如下
+-----------------+-----------------+-------------------+----------------+-------------+
| employees.name  | employees.dept  | employees.salary  | employees.age  | sum_salary  |
+-----------------+-----------------+-------------------+----------------+-------------+
| Jane            | Marketing       | 29000             | 28             | 64000       |
| Jeff            | Marketing       | 35000             | 38             | 64000       |
| Fred            | Engineering     | 21000             | 28             | 96000       |
| Tom             | Engineering     | 23000             | 33             | 96000       |
| Chloe           | Engineering     | 23000             | 25             | 96000       |
| Paul            | Engineering     | 29000             | 23             | 96000       |
| Lisa            | Sales           | 10000             | 35             | 72000       |
| Alex            | Sales           | 30000             | 33             | 72000       |
| Evan            | Sales           | 32000             | 38             | 72000       |
+-----------------+-----------------+-------------------+----------------+-------------+

结论：

  • 情形一：在partition by和order by同时存在的情况下，对于MAX | MIN | COUNT | SUM | AVG 等函数
    如果想要对相同分组中的数整体进行计算，则要显示指定RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING；
  • 情形二：如果按在分组中实现按行处理，则要显示指定ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW；
  • 情形三：默认情况下（即未显示指定window specification时）对于order by列相同的值处理结果相同。
2.1.2. 无order by下的默认值

当over语句中order by和window specification都缺失时，window specification的默认值是ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING

结论

  • 对MAX | MIN | COUNT | SUM | AVG 等函数，在order by缺失或order by和partition by相同时，效果同上述情形一
sql 复制代码 
select *,sum(salary) over(partition by dept) as sum_salary from employees;

-- 结果如下
+-----------------+-----------------+-------------------+----------------+-------------+
| employees.name  | employees.dept  | employees.salary  | employees.age  | sum_salary  |
+-----------------+-----------------+-------------------+----------------+-------------+
| Fred            | Engineering     | 21000             | 28             | 96000       |
| Tom             | Engineering     | 23000             | 33             | 96000       |
| Paul            | Engineering     | 29000             | 23             | 96000       |
| Chloe           | Engineering     | 23000             | 25             | 96000       |
| Jane            | Marketing       | 29000             | 28             | 64000       |
| Jeff            | Marketing       | 35000             | 38             | 64000       |
| Lisa            | Sales           | 10000             | 35             | 72000       |
| Evan            | Sales           | 32000             | 38             | 72000       |
| Alex            | Sales           | 30000             | 33             | 72000       |
+-----------------+-----------------+-------------------+----------------+-------------+

结果中，对于相同dept中不同员工，效果并不是按行并逐行处理，而是对相同dept下的员工进行了统一处理。即UNBOUNDED FOLLOWING表示不跟随当前处理的行，直接对中整个分区中进行计算。

疑问：ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING中rows的作用是什么了？

3. last_value函数的注意事项

前面在提到last_value时，特意强调了该函数的结果并不是分区中的最后一个值，结合上述介绍的window specification再来看下该函数的结果值。

sql 复制代码 
SELECT *,
    first_value(salary) OVER (PARTITION BY dept ORDER BY salary) AS first,
    last_value(salary) OVER (PARTITION BY dept ORDER BY salary) AS last
    FROM employees;
+-----------------+-----------------+-------------------+----------------+--------+--------+
| employees.name  | employees.dept  | employees.salary  | employees.age  | first  |  last  |
+-----------------+-----------------+-------------------+----------------+--------+--------+
| Jane            | Marketing       | 29000             | 28             | 29000  | 29000  |
| Jeff            | Marketing       | 35000             | 38             | 29000  | 35000  |
| Fred            | Engineering     | 21000             | 28             | 21000  | 21000  |
| Tom             | Engineering     | 23000             | 33             | 21000  | 23000  |
| Chloe           | Engineering     | 23000             | 25             | 21000  | 23000  |
| Paul            | Engineering     | 29000             | 23             | 21000  | 29000  |
| Lisa            | Sales           | 10000             | 35             | 10000  | 10000  |
| Alex            | Sales           | 30000             | 33             | 10000  | 30000  |
| Evan            | Sales           | 32000             | 38             | 10000  | 32000  |
+-----------------+-----------------+-------------------+----------------+--------+--------+

结果中，对于每一行的last_value的结果都是当前值，并不分区中按salary升序的最后一个值。造成这个结果的原因正是由于默认的window specification导致的（ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW）。

如想要实现得到整个分区中按salary升序的最大值，则需要显示设置window specification为RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING

sql 复制代码 
select *
,first_value(salary) over(partition by dept order by salary) as first_salary
,last_value(salary) over(partition by dept order by salary range between unbounded preceding and unbounded following) as last_salary from employees;

-- 结果如下
+-----------------+-----------------+-------------------+----------------+---------------+--------------+
| employees.name  | employees.dept  | employees.salary  | employees.age  | first_salary  | last_salary  |
+-----------------+-----------------+-------------------+----------------+---------------+--------------+
| Jane            | Marketing       | 29000             | 28             | 29000         | 35000        |
| Jeff            | Marketing       | 35000             | 38             | 29000         | 35000        |
| Fred            | Engineering     | 21000             | 28             | 21000         | 29000        |
| Tom             | Engineering     | 23000             | 33             | 21000         | 29000        |
| Chloe           | Engineering     | 23000             | 25             | 21000         | 29000        |
| Paul            | Engineering     | 29000             | 23             | 21000         | 29000        |
| Lisa            | Sales           | 10000             | 35             | 10000         | 32000        |
| Alex            | Sales           | 30000             | 33             | 10000         | 32000        |
| Evan            | Sales           | 32000             | 38             | 10000         | 32000        |
+-----------------+-----------------+-------------------+----------------+---------------+--------------+
相关推荐
让线程再跑一会1 小时前
Flink 实时数仓（七）【DWS 层搭建（一）流量域汇总表创建】
大数据·flink
胖胖学编程2 小时前
[flink]部署模式
大数据·flink
Sui_Network2 小时前
8–9月，Sui Move智能合约工作坊将在台北+线上举行
大数据·前端·人工智能·学习·区块链·智能合约
苍煜2 小时前
Flink流式框架过程问题
大数据·flink
旗晟机器人2 小时前
电力巡检机器人：助力传统能源发电的创新力量
大数据·人工智能·机器人
Tinalee-电商API接口呀3 小时前
【电商API接口项目实战分享】项目实战案例一：电商平台零售数据分析
大数据·开发语言·人工智能·python·数据挖掘·数据分析·零售
程序员不想YY啊3 小时前
【大数据开发语言Scala的入门教程】
大数据·开发语言·scala
Time_Memory_cici4 小时前
ImageNet数据集和CIFAR-10数据集
大数据·图像处理·machine learning
说私域4 小时前
信息流、资金流、物流的流转与开源 AI 智能名片 S2B2C 商城小程序的融合
大数据·人工智能·小程序
武子康5 小时前
大数据-65 Kafka 高级特性 分区 Broker自动再平衡 ISR 副本 宕机恢复再重平衡 实测
java·大数据·分布式·架构·kafka
热门推荐
01RAG 实践- Ollama+RagFlow 部署本地知识库022024电赛H题参考方案（+视频演示）——自动行使小车03Coze扣子平台完整体验和实践（附国内和国际版对比）04DSPy实战：三十分钟无痛上手自动化Prompt框架05yolov8实战第五天——yolov8+ffmpg实时视频流检测并进行实时推流——（推流，保姆教学）06组基轨迹建模 GBTM的介绍与实现（Stata 或 R）07【数学建模】2024华数杯A题思路（已完成）08带你看懂思源字体的各个版本都有什么区别09【经验分享】Ubuntu22.04安装微信(linux官方版)10【立创EDA-PCB设计基础】6.布线铺铜实战及细节详解