hive企业级调优策略之如何用Explain查看执行计划

Explain执行计划概述

Explain呈现的执行计划,由一系列Stage组成,这一系列Stage具有依赖关系,每个Stage对应一个MapReduce Job,或者一个文件系统操作等。

若某个Stage对应的一个MapReduce Job,其Map端和Reduce端的计算逻辑分别由Map Operator Tree和Reduce Operator Tree进行描述,Operator Tree由一系列的Operator组成,一个Operator代表在Map或Reduce阶段的一个单一的逻辑操作,例如TableScan Operator,Select Operator,Join Operator等。

下图是由一个执行计划绘制而成:

常见的Operator及其作用如下:

TableScan:表扫描操作,通常map端第一个操作肯定是表扫描操作

Select Operator:选取操作

Group By Operator:分组聚合操作

Reduce Output Operator:输出到 reduce 操作

Filter Operator:过滤操作

Join Operator:join 操作

File Output Operator:文件输出操作

Fetch Operator 客户端获取数据操作

基本语法

bash 复制代码
EXPLAIN [FORMATTED | EXTENDED | DEPENDENCY] query-sql

注:FORMATTED、EXTENDED、DEPENDENCY关键字为可选项,各自作用如下。

FORMATTED:将执行计划以JSON字符串的形式输出

EXTENDED:输出执行计划中的额外信息,通常是读写的文件名等信息

DEPENDENCY:输出执行计划读取的表及分区

案例实操

查看下面这条语句的执行计划

bash 复制代码
explain FORMATTED
select
    user_id,
    count(*)
from order_detail
group by user_id;
bash 复制代码
{"STAGE DEPENDENCIES":{"Stage-1":{"ROOT STAGE":"TRUE"},"Stage-0":{"DEPENDENT STAGES":"Stage-1"}},"STAGE PLANS":{"Stage-1":{"Map Reduce":{"Map Operator Tree:":[{"TableScan":{"alias:":"order_detail","columns:":["user_id"],"database:":"demodata","Statistics:":"Num rows: 13066777 Data size: 11760099340 Basic stats: COMPLETE Column stats: NONE","table:":"order_detail","isTempTable:":"false","OperatorId:":"TS_0","children":{"Select Operator":{"expressions:":"user_id (type: string)","columnExprMap:":{"BLOCK__OFFSET__INSIDE__FILE":"BLOCK__OFFSET__INSIDE__FILE","INPUT__FILE__NAME":"INPUT__FILE__NAME","ROW__ID":"ROW__ID","create_time":"create_time","dt":"dt","id":"id","product_id":"product_id","product_num":"product_num","province_id":"province_id","total_amount":"total_amount","user_id":"user_id"},"outputColumnNames:":["user_id"],"Statistics:":"Num rows: 13066777 Data size: 11760099340 Basic stats: COMPLETE Column stats: NONE","OperatorId:":"SEL_7","children":{"Group By Operator":{"aggregations:":["count()"],"columnExprMap:":{"_col0":"user_id"},"keys:":"user_id (type: string)","mode:":"hash","outputColumnNames:":["_col0","_col1"],"Statistics:":"Num rows: 13066777 Data size: 11760099340 Basic stats: COMPLETE Column stats: NONE","OperatorId:":"GBY_8","children":{"Reduce Output Operator":{"columnExprMap:":{"KEY._col0":"_col0","VALUE._col0":"_col1"},"key expressions:":"_col0 (type: string)","sort order:":"+","Map-reduce partition columns:":"_col0 (type: string)","Statistics:":"Num rows: 13066777 Data size: 11760099340 Basic stats: COMPLETE Column stats: NONE","value expressions:":"_col1 (type: bigint)","OperatorId:":"RS_9"}}}}}}}}],"Execution mode:":"vectorized","Reduce Operator Tree:":{"Group By Operator":{"aggregations:":["count(VALUE._col0)"],"columnExprMap:":{"_col0":"KEY._col0"},"keys:":"KEY._col0 (type: string)","mode:":"mergepartial","outputColumnNames:":["_col0","_col1"],"Statistics:":"Num rows: 6533388 Data size: 5880049219 Basic stats: COMPLETE Column stats: NONE","OperatorId:":"GBY_4","children":{"File Output Operator":{"compressed:":"false","Statistics:":"Num rows: 6533388 Data size: 5880049219 Basic stats: COMPLETE Column stats: NONE","table:":{"input format:":"org.apache.hadoop.mapred.SequenceFileInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"},"OperatorId:":"FS_6"}}}}}},"Stage-0":{"Fetch Operator":{"limit:":"-1","Processor Tree:":{"ListSink":{"OperatorId:":"LIST_SINK_10"}}}}}}

执行计划如下图

相关推荐
爱吃面的猫17 小时前
大数据Hadoop之——Flink1.17.0安装与使用(非常详细)
大数据·hadoop·分布式
zhuiQiuMX18 小时前
脉脉maimai面试死亡日记
数据仓库·sql·面试
Edingbrugh.南空1 天前
Hadoop MapReduce 入门
大数据·hadoop·mapreduce
大数据CLUB2 天前
基于spark的奥运会奖牌变化数据分析
大数据·hadoop·数据分析·spark
Edingbrugh.南空2 天前
Hadoop高可用集群搭建
大数据·hadoop·分布式
无级程序员3 天前
hive2服务启动报错:/tmp/hive on HDFS should be writable(不是chmod 777能解决的)
hive·hadoop·hdfs
rui锐rui3 天前
大数据学习2:HIve
大数据·hive·学习
凌辰揽月3 天前
Servlet学习
hive·学习·servlet
weixin_307779134 天前
Hive集群之间迁移的Linux Shell脚本
大数据·linux·hive·bash·迁移学习
王小王-1234 天前
基于Hadoop的公共自行车数据分布式存储和计算平台的设计与实现
大数据·hive·hadoop·分布式·hadoop公共自行车·共享单车大数据分析·hadoop共享单车