Hive:数据仓库利器

1. 简介

Hive是一个基于Hadoop的开源数据仓库工具,可以用来存储、查询和分析大规模数据。Hive使用SQL-like的HiveQL语言来查询数据,并将其结果存储在Hadoop的文件系统中。

2. 基本概念

介绍 Hive 的核心概念,例如表、分区、桶、HQL 等。

2.1 架构

Design - Apache Hive - Apache Software Foundation

|----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 组成 | 详情 |
| UI | The user interfacefor users to submit queries and other operations to the system. As of 2011 the system had a command line interface and a web based GUI was being developed. |
| Driver | The component whichreceives the queries. This component implements the notion of session handles and provides execute and fetch APIs modeled on JDBC/ODBC interfaces. |
| Compiler | The component thatparses the query, does semantic analysis on the different query blocks and query expressions and eventually generates an execution plan with the help of the table and partition metadata looked up from the metastore. |
| Metastore | The component that stores all the structure information of the various tables and partitions in the warehouse including column and column type information, the serializers and deserializers necessary to read and write data and the corresponding HDFS files where the data is stored. |
| Execution Engine | The component which executes the execution plan created by the compiler. The plan is a DAG of stages. The execution engine manages the dependencies between these different stages of the plan and executes these stages on the appropriate system components. |

2.2 Data Model

|----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 类型 | 详情 |
| Tables | These are analogous to Tables in Relational Databases. Tables can be filtered, projected, joined and unioned. Additionally all the data of a table is stored in a directory in HDFS. Hive also supports the notion of external tables wherein a table can be created on prexisting files or directories in HDFS by providing the appropriate location to the table creation DDL. The rows in a table are organized into typed columns similar to Relational Databases. |
| Partitions | Each Table can have one or more partition keys which determine how the data is stored, for example a table T with a date partition column ds had files with data for a particular date stored in the <table location>/ds=<date> directory in HDFS. Partitions allow the system to prune data to be inspected based on query predicates, for example a query that is interested in rows from T that satisfy the predicate T.ds = '2008-09-01' would only have to look at files in <table location>/ds=2008-09-01/ directory in HDFS. |
| Buckets | **Data in each partition may in turn be divided into Buckets based on the hash of a column in the table.**Each bucket is stored as a file in the partition directory. Bucketing allows the system to efficiently evaluate queries that depend on a sample of data (these are queries that use the SAMPLE clause on the table). |

3. 实践应用

3.1 数仓建设

4. 性能优化

介绍如何优化 Hive 的性能

5. 常见问题解答

5.1 常用SQL

|--------|--------------------------------|
| 场景 | SQL |
| 连续n天登录 | sql SELECT * FROM test; |
| | |

6. 总结

总结 Hive 的关键知识点,并提供学习资源和进一步研究方向。

相关推荐
Lx3527 小时前
Hadoop容错机制深度解析:保障作业稳定运行
大数据·hadoop
IT毕设梦工厂14 小时前
大数据毕业设计选题推荐-基于大数据的客户购物订单数据分析与可视化系统-Hadoop-Spark-数据可视化-BigData
大数据·hadoop·数据分析·spark·毕业设计·源码·bigdata
大数据CLUB21 小时前
基于spark的澳洲光伏发电站选址预测
大数据·hadoop·分布式·数据分析·spark·数据开发
计算机编程小央姐1 天前
跟上大数据时代步伐:食物营养数据可视化分析系统技术前沿解析
大数据·hadoop·信息可视化·spark·django·课程设计·食物
IT学长编程1 天前
计算机毕业设计 基于Hadoop的健康饮食推荐系统的设计与实现 Java 大数据毕业设计 Hadoop毕业设计选题【附源码+文档报告+安装调试】
java·大数据·hadoop·毕业设计·课程设计·推荐算法·毕业论文
Lx3521 天前
Hadoop数据一致性保障:处理分布式系统常见问题
大数据·hadoop
IT学长编程1 天前
计算机毕业设计 基于Hadoop豆瓣电影数据可视化分析设计与实现 Python 大数据毕业设计 Hadoop毕业设计选题【附源码+文档报告+安装调试
大数据·hadoop·python·django·毕业设计·毕业论文·豆瓣电影数据可视化分析
Dobby_051 天前
【Hadoop】Yarn:Hadoop 生态的资源操作系统
大数据·hadoop·分布式·yarn
笨蛋少年派1 天前
安装Hadoop中遇到的一些问题和解决
大数据·hadoop·分布式