记录一个hive中因没启yarn导致的spark引擎跑insert语句的报错

【背景说明】

刚在hive中配置了Spark引擎,在进行Hive on Spark测试时报错,

报错截图如下:

复制代码
[atguigu@hadoop102 conf]$ hive
which: no hbase in (/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/module/jdk1.8.0_212/bin:/opt/module/hadoop-3.3.4/bin:/opt/module/hadoop-3.3.4/sbin:/opt/module/hive-3.1.3/bin                          :/opt/module/kafka/bin:/opt/module/efak/bin:/home/atguigu/.local/bin:/home/atguigu/bin:/opt/module/jdk1.8.0_212/bin:/opt/module/hadoop-3.3.4/bin:/opt/module/hadoop-3.3.4/sbin:/opt/modu                          le/hive-3.1.3/bin:/opt/module/kafka/bin:/opt/module/efak/bin:/opt/module/spark/bin)
Hive Session ID = 4b43a439-6dee-4295-a467-7182adb64f04

Logging initialized using configuration in file:/opt/module/hive-3.1.3/conf/hive-log4j2.properties Async: true
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of                           the driver class is generally unnecessary.
Hive Session ID = 6dbba42a-f926-4cee-8368-646383608b57
hive (default)> create table student(id int, name string);
OK
Time taken: 0.948 seconds
hive (default)> insert into table student values(1,'abc');
Query ID = atguigu_20240420093653_68ffa538-97fa-4864-9d92-18dfc9def1c6
Total jobs = 1
Launching Job 1 out of 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark client for Spark session 885f9da9-d447-4d55-a411-aca9c832703b)'
FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session 885f9da9-d447-4d55-a411-aca9c832703b

Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark client for Spark session 885f9da9-d447-4d55-a411-aca9c832703b)'
FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session 885f9da9-d447-4d55-a411-aca9c832703b

【原因】

百度说是这个报错意味着Hive无法为Spark会话创建Spark客户端。可能是由于配置问题导致的。建议检查Hive配置文件中关于Spark的设置是否正确,特别是关于Spark执行引擎的配置。

【解决】

这次没有创建SparkClient失败是因为我的yarn没启,Spark运行需要yarn进行资源调度。好,启动yarn:start-yarn.sh

再跑:hive (default)> insert into table student values(1,'abc');

相关推荐
DataY轻量数据平台2 小时前
任务调度的三种模式:什么时候跑、按什么顺序跑、等谁跑完
数据仓库·etl
演艺大赏5 小时前
三个坑连环套:一次 NameNode 起不来,从 JournalNode 未格式化查到配置里多了个挂载点
大数据·hadoop
宽海智能仓储物流7 小时前
宽海智能WCS系统常见问题全解析:从选型对接、运维定制到降本增效
运维·数据仓库·自动化
starzy19909 小时前
Hadoop介绍总结:一张图理清组件、生态与版本
java·hadoop
演艺大赏1 天前
两个 NameNode 一起起不来以及Hadoop 3.3.2 里那个认不出自家 JournalNode 的 Servlet
大数据·hadoop
演艺大赏1 天前
RESTServer 起不来,日志只有一行:headerCacheSize 65536 exceeds maximum value of 65534
大数据·hadoop·hbase
吾AI科技3 天前
基于Tez引擎的 Hive SQL 性能优化 —— 为什么脚本运行越来越慢
大数据·hive·性能优化·tez
上海潭龙东海科技有限公司4 天前
客运站管控之动态孪生技术方案(新增一屏管控模块
数据仓库·人工智能·算法·重构·动态规划
freejackman8 天前
【工程化】从“单兵作战”到“多引擎架构”:基于门面模式的重构实战
大数据·数据库·hive·sql·测试工具·设计模式·架构
skywalk816311 天前
hive 常驻集群:像 OSPF 一样在线,像 git 一样可信
运维·人工智能·hive