hive企业级调优策略之如何用Explain查看执行计划

Explain执行计划概述

Explain呈现的执行计划,由一系列Stage组成,这一系列Stage具有依赖关系,每个Stage对应一个MapReduce Job,或者一个文件系统操作等。

若某个Stage对应的一个MapReduce Job,其Map端和Reduce端的计算逻辑分别由Map Operator Tree和Reduce Operator Tree进行描述,Operator Tree由一系列的Operator组成,一个Operator代表在Map或Reduce阶段的一个单一的逻辑操作,例如TableScan Operator,Select Operator,Join Operator等。

下图是由一个执行计划绘制而成:

常见的Operator及其作用如下:

TableScan:表扫描操作,通常map端第一个操作肯定是表扫描操作

Select Operator:选取操作

Group By Operator:分组聚合操作

Reduce Output Operator:输出到 reduce 操作

Filter Operator:过滤操作

Join Operator:join 操作

File Output Operator:文件输出操作

Fetch Operator 客户端获取数据操作

基本语法

bash 复制代码
EXPLAIN [FORMATTED | EXTENDED | DEPENDENCY] query-sql

注:FORMATTED、EXTENDED、DEPENDENCY关键字为可选项,各自作用如下。

FORMATTED:将执行计划以JSON字符串的形式输出

EXTENDED:输出执行计划中的额外信息,通常是读写的文件名等信息

DEPENDENCY:输出执行计划读取的表及分区

案例实操

查看下面这条语句的执行计划

bash 复制代码
explain FORMATTED
select
    user_id,
    count(*)
from order_detail
group by user_id;
bash 复制代码
{"STAGE DEPENDENCIES":{"Stage-1":{"ROOT STAGE":"TRUE"},"Stage-0":{"DEPENDENT STAGES":"Stage-1"}},"STAGE PLANS":{"Stage-1":{"Map Reduce":{"Map Operator Tree:":[{"TableScan":{"alias:":"order_detail","columns:":["user_id"],"database:":"demodata","Statistics:":"Num rows: 13066777 Data size: 11760099340 Basic stats: COMPLETE Column stats: NONE","table:":"order_detail","isTempTable:":"false","OperatorId:":"TS_0","children":{"Select Operator":{"expressions:":"user_id (type: string)","columnExprMap:":{"BLOCK__OFFSET__INSIDE__FILE":"BLOCK__OFFSET__INSIDE__FILE","INPUT__FILE__NAME":"INPUT__FILE__NAME","ROW__ID":"ROW__ID","create_time":"create_time","dt":"dt","id":"id","product_id":"product_id","product_num":"product_num","province_id":"province_id","total_amount":"total_amount","user_id":"user_id"},"outputColumnNames:":["user_id"],"Statistics:":"Num rows: 13066777 Data size: 11760099340 Basic stats: COMPLETE Column stats: NONE","OperatorId:":"SEL_7","children":{"Group By Operator":{"aggregations:":["count()"],"columnExprMap:":{"_col0":"user_id"},"keys:":"user_id (type: string)","mode:":"hash","outputColumnNames:":["_col0","_col1"],"Statistics:":"Num rows: 13066777 Data size: 11760099340 Basic stats: COMPLETE Column stats: NONE","OperatorId:":"GBY_8","children":{"Reduce Output Operator":{"columnExprMap:":{"KEY._col0":"_col0","VALUE._col0":"_col1"},"key expressions:":"_col0 (type: string)","sort order:":"+","Map-reduce partition columns:":"_col0 (type: string)","Statistics:":"Num rows: 13066777 Data size: 11760099340 Basic stats: COMPLETE Column stats: NONE","value expressions:":"_col1 (type: bigint)","OperatorId:":"RS_9"}}}}}}}}],"Execution mode:":"vectorized","Reduce Operator Tree:":{"Group By Operator":{"aggregations:":["count(VALUE._col0)"],"columnExprMap:":{"_col0":"KEY._col0"},"keys:":"KEY._col0 (type: string)","mode:":"mergepartial","outputColumnNames:":["_col0","_col1"],"Statistics:":"Num rows: 6533388 Data size: 5880049219 Basic stats: COMPLETE Column stats: NONE","OperatorId:":"GBY_4","children":{"File Output Operator":{"compressed:":"false","Statistics:":"Num rows: 6533388 Data size: 5880049219 Basic stats: COMPLETE Column stats: NONE","table:":{"input format:":"org.apache.hadoop.mapred.SequenceFileInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"},"OperatorId:":"FS_6"}}}}}},"Stage-0":{"Fetch Operator":{"limit:":"-1","Processor Tree:":{"ListSink":{"OperatorId:":"LIST_SINK_10"}}}}}}

执行计划如下图

相关推荐
计算机编程-吉哥9 小时前
大数据毕业设计-基于Python的中文起点网小说数据分析平台(高分计算机毕业设计选题·定制开发·真正大数据)
大数据·hadoop·计算机毕业设计选题·机器学习毕业设计·大数据毕业设计·大数据毕业设计选题推荐·大数据毕设项目
Agatha方艺璇12 小时前
Hive基础简介
数据仓库·hive·hadoop
IT研究室14 小时前
大数据毕业设计选题推荐-基于大数据的国内旅游景点游客数据分析系统-Spark-Hadoop-Bigdata
大数据·hadoop·spark·毕业设计·源码·数据可视化·bigdata
Lx35215 小时前
YARN资源调度优化:最大化集群利用率
大数据·hadoop
Leo.yuan15 小时前
不同数据仓库模型有什么不同?企业如何选择适合的数据仓库模型?
大数据·数据库·数据仓库·信息可视化·spark
chat2tomorrow16 小时前
数据采集平台的起源与演进:从ETL到数据复制
大数据·数据库·数据仓库·mysql·低代码·postgresql·etl
isfox1 天前
Google GFS 深度解析:分布式文件系统的开山之作
大数据·hadoop
鼠鼠我捏,要死了捏1 天前
Hadoop NameNode内存泄漏与GC停顿问题排查与解决方案
hadoop·问题排查·jvm优化
嘉禾望岗5032 天前
Yarn介绍与HA搭建
大数据·hadoop·yarn