大数据开发-Hive介绍以及安装配置

拉霍拉卡2024-03-12 8:13

文章目录

数据库和数据仓库的区别

  • 数据库：传统的关系型数据库主要应用在基本的事务处理，比如交易，支持增删改查
  • 数据仓库：主要做一些复杂的分析操作，侧重决策支持，相对于数据库而言，数据仓库分析的数据规模要大的多，只支持查询
  • 本质区别是OLTP(On-Line-Transaction Processing)和OLAP(On-Line-Analytical Processing)的区别，OLTP称为联机事务处理，也是面向交易的处理系统，它是针对具体的业务在数据库联机的日常操作，通常对少数记录进行查询、修改，用户关心的是响应；时间，数据的安全性，完整性等问题;OLAP是分析性处理，称为联机分析处理，一般针对某些主题历史数据进行分析，支持管理决策

Hive安装配置

shell 复制代码 
# 解压完之后
[root@hadoop04 conf]# mv hive-env.sh.template hive-env.sh
[root@hadoop04 conf]# mv hive-default.xml.template hive-site.xml

#修改配置
[root@hadoop04 conf]# vim hive-env.sh
export JAVA_HOME=/home/soft/jdk1.8
export HIVE_HOME=/home/soft/apache-hive-3.1.2
export HADOOP_HOME=/home/soft/hadoop-3.2.0

# 根据name修改对应配置
[root@hadoop04 conf]# vim hive-site.xml 
 </property>
  <property>
    <name>hive.exec.local.scratchdir</name>
    <value>/home/hive_repo/scratchdir</value>
    <description>Local scratch space for Hive jobs</description>
  </property>
  <property>
    <name>hive.downloaded.resources.dir</name>
    <value>/home/hive_repo/resources</value>
    <description>Temporary local directory for added resources in the remote file system.</description>
  </property>
  
    <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://ip:port/hive?serverTimezone=Asia/Shanghai</value>
    <description>
      JDBC connect string for a JDBC metastore.
      To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
      For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
    </description>
  </property>
  
    <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>root</value>
    <description>Username to use against metastore database</description>
  </property>
  
    <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>123456</value>
    <description>password to use against metastore database</description>
  </property>
    <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.cj.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
  </property>
  
  # 初始化数据仓库
  [root@hadoop04 apache-hive-3.1.2]# bin/schematool -dbType mysql -initSchema

# 看到有下面那些表就算完成啦

Hive使用方式

命令行方式

shell 复制代码 
# 连接hive
[root@hadoop04 apache-hive-3.1.2]# bin/hive
which: no hbase in (/home/soft/jdk1.8/bin:/home/soft/hadoop-3.2.0/bin:/home/soft/hadoop-3.2.0/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/soft/apache-hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/soft/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 505baa88-4bd1-4f00-9345-448ae17ab151

Logging initialized using configuration in jar:file:/home/soft/apache-hive-3.1.2/lib/hive-common-3.1.2.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Hive Session ID = dfffb77e-23d3-4c56-9457-32f30b5f4e3c

# 查询
hive> show tables;
OK
Time taken: 1.019 seconds
# 建表
hive> create table t1(id int,name string);
OK
Time taken: 1.875 seconds
hive> show tables;
OK
t1
Time taken: 0.388 seconds, Fetched: 1 row(s)
# 插入数据 会进行mapreduce
hive> insert into t1(id,name)values(1,"test");
Query ID = root_20240311140339_1e1450d1-2227-4b3d-bb10-e21f0016903b
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Starting Job = job_1710135432246_0001, Tracking URL = http://hadoop01:8088/proxy/application_1710135432246_0001/
Kill Command = /home/soft/hadoop-3.2.0/bin/mapred job  -kill job_1710135432246_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2024-03-11 14:04:00,036 Stage-1 map = 0%,  reduce = 0%
2024-03-11 14:04:08,605 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.66 sec
2024-03-11 14:04:16,949 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 4.69 sec
MapReduce Total cumulative CPU time: 4 seconds 690 msec
Ended Job = job_1710135432246_0001
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to directory hdfs://hadoop01:9000/user/hive/warehouse/t1/.hive-staging_hive_2024-03-11_14-03-39_724_266361142260875320-1/-ext-10000
Loading data to table default.t1
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 1  Reduce: 1   Cumulative CPU: 4.69 sec   HDFS Read: 15158 HDFS Write: 236 SUCCESS
Total MapReduce CPU Time Spent: 4 seconds 690 msec
OK
Time taken: 42.328 seconds
hive> select * from t1;
OK
1	test
Time taken: 0.726 seconds, Fetched: 1 row(s)
hive> drop table t1;
OK
Time taken: 1.368 seconds
# 退出
hive> quit;

Hive日志配置

运行时日志

shell 复制代码 
[root@hadoop04 conf]# mv hive-log4j2.properties.template hive-log4j2.properties
[root@hadoop04 conf]# vim hive-log4j2.properties 
# list of properties
property.hive.log.level = INFO
property.hive.root.logger = DRFA
property.hive.log.dir = /home/hive_repo/log
property.hive.log.file = hive.log
property.hive.perflogger.log.level = INFO

任务执行日志

shell 复制代码 
[root@hadoop04 conf]# mv hive-exec-log4j2.properties.template hive-exec-log4j2.properties
[root@hadoop04 conf]# vim hive-exec-log4j2.properties 

status = INFO
name = HiveExecLog4j2
packages = org.apache.hadoop.hive.ql.log

# list of properties
property.hive.log.level = INFO
property.hive.root.logger = FA
property.hive.query.id = hadoop
property.hive.log.dir = /home/hive_repo/log
property.hive.log.file = ${sys:hive.query.id}.log

level = INFO

property.hive.root.logger = FA

property.hive.query.id = hadoop

property.hive.log.dir = /home/hive_repo/log

property.hive.log.file = ${sys:hive.query.id}.log

复制代码
相关推荐
在下不上天13 分钟前
Flume日志采集系统的部署,实现flume负载均衡，flume故障恢复
大数据·开发语言·python
智慧化智能化数字化方案42 分钟前
华为IPD流程管理体系L1至L5最佳实践-解读
大数据·华为
PersistJiao2 小时前
在 Spark RDD 中，sortBy 和 top 算子的各自适用场景
大数据·spark·top·sortby
2301_811274312 小时前
大数据基于Spring Boot的化妆品推荐系统的设计与实现
大数据·spring boot·后端
Yz98762 小时前
hive的存储格式
大数据·数据库·数据仓库·hive·hadoop·数据库开发
青云交2 小时前
大数据新视界 -- 大数据大厂之 Hive 数据导入：多源数据集成的策略与实战（上）（3/ 30）
大数据·数据清洗·电商数据·数据整合·hive 数据导入·多源数据·影视娱乐数据
lzhlizihang2 小时前
python如何使用spark操作hive
hive·python·spark
武子康2 小时前
大数据-230 离线数仓 - ODS层的构建 Hive处理 UDF 与 SerDe 处理 与 当前总结
java·大数据·数据仓库·hive·hadoop·sql·hdfs
武子康2 小时前
大数据-231 离线数仓 - DWS 层、ADS 层的创建 Hive 执行脚本
java·大数据·数据仓库·hive·hadoop·mysql
时差9533 小时前
Flink Standalone集群模式安装部署
大数据·分布式·flink·部署
热门推荐
01(欧拉)openEuler系统添加网卡文件配置流程、(欧拉)openEuler系统手动配置ipv6地址流程、(欧拉)openEuler系统网络管理说明02PyTorch机器学习实现液态神经网络03【HarmonyOS】HUAWEI DevEco Studio 下载地址汇总04玄机平台应急响应—webshell查杀05Coze扣子平台完整体验和实践（附国内和国际版对比）06Ubuntu 20.04使用Livox mid 360 测试 FAST_LIO07【目标跟踪】相机运动补偿08Unity中PICO实现 隔空取物 和 接触抓取物体09RAG 实践- Ollama+RagFlow 部署本地知识库10量化方法怎么选？如何评估量化后的大模型LLM？