hive 谓词下推实例分析(on与where的区别)

测试数据

  • t1 表
bash 复制代码
select * from t1;
+--------+----------+---------+--------+
| t1.id  | t1.name  | t1.age  | t1.dt  |
+--------+----------+---------+--------+
| 1      | aa       | 12      | 01     |
| 1      | aa       | 12      | 02     |
| 2      | aa       | 14      | 01     |
| 2      | bb       | 14      | 02     |
| 3      | cc       | 16      | 02     |
| NULL   | aa       | 12      | 01     |
+--------+----------+---------+--------+
  • t2 表
bash 复制代码
select * from t2;
+--------+----------+---------+--------+
| t2.id  | t2.name  | t2.age  | t2.dt  |
+--------+----------+---------+--------+
| 1      | 1        | aa      | 12     |
| 2      | 1        | aa      | 12     |
| 1      | NULL     | aa      | 12     |
| 1      | 2        | aa      | 14     |
| 2      | 2        | bb      | 14     |
| 2      | 3        | cc      | 16     |
+--------+----------+---------+--------+

关联查询

  • t1 left join t2
bash 复制代码
select * from t1 a left join t2 b on a.id=b.id;
+-------+---------+--------+-------+-------+---------+--------+-------+
| a.id  | a.name  | a.age  | a.dt  | b.id  | b.name  | b.age  | b.dt  |
+-------+---------+--------+-------+-------+---------+--------+-------+
| 3     | cc      | 16     | 02    | NULL  | NULL    | NULL   | NULL  |
| NULL  | aa      | 12     | 01    | NULL  | NULL    | NULL   | NULL  |
| 1     | aa      | 12     | 01    | 1     | NULL    | aa     | 12    |
| 1     | aa      | 12     | 01    | 1     | 1       | aa     | 12    |
| 1     | aa      | 12     | 01    | 1     | 2       | aa     | 14    |
| 1     | aa      | 12     | 02    | 1     | NULL    | aa     | 12    |
| 1     | aa      | 12     | 02    | 1     | 1       | aa     | 12    |
| 1     | aa      | 12     | 02    | 1     | 2       | aa     | 14    |
| 2     | aa      | 14     | 01    | 2     | 3       | cc     | 16    |
| 2     | aa      | 14     | 01    | 2     | 1       | aa     | 12    |
| 2     | aa      | 14     | 01    | 2     | 2       | bb     | 14    |
| 2     | bb      | 14     | 02    | 2     | 3       | cc     | 16    |
| 2     | bb      | 14     | 02    | 2     | 1       | aa     | 12    |
| 2     | bb      | 14     | 02    | 2     | 2       | bb     | 14    |
+-------+---------+--------+-------+-------+---------+--------+-------+
  • t1 left join t2 on a.id=b.id where a.dt ='01',保留表谓词下推,map端提前过滤
bash 复制代码
select * from t1 a left join t2 b on a.id=b.id where a.dt ='01';
+-------+---------+--------+-------+-------+---------+--------+-------+
| a.id  | a.name  | a.age  | a.dt  | b.id  | b.name  | b.age  | b.dt  |
+-------+---------+--------+-------+-------+---------+--------+-------+
| NULL  | aa      | 12     | 01    | NULL  | NULL    | NULL   | NULL  |
| 1     | aa      | 12     | 01    | 1     | NULL    | aa     | 12    |
| 1     | aa      | 12     | 01    | 1     | 1       | aa     | 12    |
| 1     | aa      | 12     | 01    | 1     | 2       | aa     | 14    |
| 2     | aa      | 14     | 01    | 2     | 3       | cc     | 16    |
| 2     | aa      | 14     | 01    | 2     | 1       | aa     | 12    |
| 2     | aa      | 14     | 01    | 2     | 2       | bb     | 14    |
+-------+---------+--------+-------+-------+---------+--------+-------+
  • t1 left join t2 on a.id=b.id and a.dt ='01',保留表非谓词下推,reduce端过滤
bash 复制代码
select * from t1 a left join t2 b on a.id=b.id and a.dt ='01';
+-------+---------+--------+-------+-------+---------+--------+-------+
| a.id  | a.name  | a.age  | a.dt  | b.id  | b.name  | b.age  | b.dt  |
+-------+---------+--------+-------+-------+---------+--------+-------+
| 1     | aa      | 12     | 02    | NULL  | NULL    | NULL   | NULL  |
| 2     | bb      | 14     | 02    | NULL  | NULL    | NULL   | NULL  |
| 3     | cc      | 16     | 02    | NULL  | NULL    | NULL   | NULL  |
| NULL  | aa      | 12     | 01    | NULL  | NULL    | NULL   | NULL  |
| 1     | aa      | 12     | 01    | 1     | 2       | aa     | 14    |
| 1     | aa      | 12     | 01    | 1     | NULL    | aa     | 12    |
| 1     | aa      | 12     | 01    | 1     | 1       | aa     | 12    |
| 2     | aa      | 14     | 01    | 2     | 2       | bb     | 14    |
| 2     | aa      | 14     | 01    | 2     | 3       | cc     | 16    |
| 2     | aa      | 14     | 01    | 2     | 1       | aa     | 12    |
+-------+---------+--------+-------+-------+---------+--------+-------+

说明:保留表应用谓词下推,提前过滤,会把不符合条件的数据提前过滤掉;保留表不应用谓词下推,不提前过滤,只能在join发生时,不符合条件的数据不参与关联计算;

  • t1 left join t2 on a.id=b.id and b.dt ='12',空表谓词下推,map端过滤
bash 复制代码
select * from t1 a left join t2 b on a.id=b.id and b.dt ='12';
+-------+---------+--------+-------+-------+---------+--------+-------+
| a.id  | a.name  | a.age  | a.dt  | b.id  | b.name  | b.age  | b.dt  |
+-------+---------+--------+-------+-------+---------+--------+-------+
| 2     | aa      | 14     | 01    | 2     | 1       | aa     | 12    |
| 2     | bb      | 14     | 02    | 2     | 1       | aa     | 12    |
| 3     | cc      | 16     | 02    | NULL  | NULL    | NULL   | NULL  |
| NULL  | aa      | 12     | 01    | NULL  | NULL    | NULL   | NULL  |
| 1     | aa      | 12     | 01    | 1     | NULL    | aa     | 12    |
| 1     | aa      | 12     | 01    | 1     | 1       | aa     | 12    |
| 1     | aa      | 12     | 02    | 1     | NULL    | aa     | 12    |
| 1     | aa      | 12     | 02    | 1     | 1       | aa     | 12    |
+-------+---------+--------+-------+-------+---------+--------+-------+
  • t1 left join t2 on a.id=b.id where b.dt ='12',空表非谓词下推,reduce端过滤
bash 复制代码
select * from t1 a left join t2 b on a.id=b.id where b.dt ='12';
+-------+---------+--------+-------+-------+---------+--------+-------+
| a.id  | a.name  | a.age  | a.dt  | b.id  | b.name  | b.age  | b.dt  |
+-------+---------+--------+-------+-------+---------+--------+-------+
| 2     | aa      | 14     | 01    | 2     | 1       | aa     | 12    |
| 2     | bb      | 14     | 02    | 2     | 1       | aa     | 12    |
| 1     | aa      | 12     | 01    | 1     | NULL    | aa     | 12    |
| 1     | aa      | 12     | 01    | 1     | 1       | aa     | 12    |
| 1     | aa      | 12     | 02    | 1     | NULL    | aa     | 12    |
| 1     | aa      | 12     | 02    | 1     | 1       | aa     | 12    |
+-------+---------+--------+-------+-------+---------+--------+-------+

说明:空表应用谓词下推,提前过滤,会把不符合条件的数据提前过滤掉;空表不应用谓词下推,不提前过滤,只能在join完成时,过滤不符合条件的数据;

总结 : 是否应用谓词下推,最后产生的结果往往不同,这里需要特别注意。

参考:
一文弄懂Hive中谓词下推(on与where的区别)

相关推荐
星落梦生1 小时前
SpringAOP基础笔记(7)
java·开发语言
金銀銅鐵2 小时前
带您看懂字节码指令(四):一条指令里有什么
java·jvm
最强小杰2 小时前
某 OpenAI 兼容网关上的 `kimi-k3.1-pro`:Cline、Zed 接入与 SSE 排错示例
java·服务器·网络·ai
Flynn Liang3 小时前
流量治理不是限流那么简单:I/O 密集型 Java 服务的 6 层防线
java·微服务·高并发·限流·熔断·系统稳定性·resilience4j
学长毕设3 小时前
基于SpringBoot智慧养老服务平台设计与实现-附源码
java·spring boot·后端
曹牧3 小时前
Java:“刷新雪崩”
java·开发语言
秋天的怀念_David3 小时前
算法一千零一夜 · 第三十三夜 · 跳表
java·程序人生·算法
Forget the Dream5 小时前
Spring AI 2.0 系列(二):第一个 ChatClient——从注入到一次完整调用
java·人工智能·spring boot·ai·springai
黑十6 小时前
FDE 怎么快速搭一个可演示页面?最小前端方案
java·大数据·前端·数据库·人工智能
神明不懂浪漫7 小时前
【第八章】多线程(一)
java·开发语言·经验分享·笔记