记录一个hive中因没启yarn导致的spark引擎跑insert语句的报错

【背景说明】

刚在hive中配置了Spark引擎,在进行Hive on Spark测试时报错,

报错截图如下:

复制代码
[atguigu@hadoop102 conf]$ hive
which: no hbase in (/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/module/jdk1.8.0_212/bin:/opt/module/hadoop-3.3.4/bin:/opt/module/hadoop-3.3.4/sbin:/opt/module/hive-3.1.3/bin                          :/opt/module/kafka/bin:/opt/module/efak/bin:/home/atguigu/.local/bin:/home/atguigu/bin:/opt/module/jdk1.8.0_212/bin:/opt/module/hadoop-3.3.4/bin:/opt/module/hadoop-3.3.4/sbin:/opt/modu                          le/hive-3.1.3/bin:/opt/module/kafka/bin:/opt/module/efak/bin:/opt/module/spark/bin)
Hive Session ID = 4b43a439-6dee-4295-a467-7182adb64f04

Logging initialized using configuration in file:/opt/module/hive-3.1.3/conf/hive-log4j2.properties Async: true
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of                           the driver class is generally unnecessary.
Hive Session ID = 6dbba42a-f926-4cee-8368-646383608b57
hive (default)> create table student(id int, name string);
OK
Time taken: 0.948 seconds
hive (default)> insert into table student values(1,'abc');
Query ID = atguigu_20240420093653_68ffa538-97fa-4864-9d92-18dfc9def1c6
Total jobs = 1
Launching Job 1 out of 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark client for Spark session 885f9da9-d447-4d55-a411-aca9c832703b)'
FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session 885f9da9-d447-4d55-a411-aca9c832703b

Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark client for Spark session 885f9da9-d447-4d55-a411-aca9c832703b)'
FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session 885f9da9-d447-4d55-a411-aca9c832703b

【原因】

百度说是这个报错意味着Hive无法为Spark会话创建Spark客户端。可能是由于配置问题导致的。建议检查Hive配置文件中关于Spark的设置是否正确,特别是关于Spark执行引擎的配置。

【解决】

这次没有创建SparkClient失败是因为我的yarn没启,Spark运行需要yarn进行资源调度。好,启动yarn:start-yarn.sh

再跑:hive (default)> insert into table student values(1,'abc');

相关推荐
大数据CLUB4 小时前
基于spark的奥运会奖牌变化数据分析
大数据·hadoop·数据分析·spark
Edingbrugh.南空4 小时前
Hadoop高可用集群搭建
大数据·hadoop·分布式
无级程序员1 天前
hive2服务启动报错:/tmp/hive on HDFS should be writable(不是chmod 777能解决的)
hive·hadoop·hdfs
rui锐rui1 天前
大数据学习2:HIve
大数据·hive·学习
凌辰揽月1 天前
Servlet学习
hive·学习·servlet
weixin_307779132 天前
Hive集群之间迁移的Linux Shell脚本
大数据·linux·hive·bash·迁移学习
王小王-1232 天前
基于Hadoop的公共自行车数据分布式存储和计算平台的设计与实现
大数据·hive·hadoop·分布式·hadoop公共自行车·共享单车大数据分析·hadoop共享单车
王小王-1232 天前
基于Hadoop的大规模文本词频统计分析系统设计与实现
hadoop·mapreduce·hadoop词频统计·hadoop文本统计·mapreduce词频统计
陈敬雷-充电了么-CEO兼CTO2 天前
推荐算法系统系列>推荐数据仓库集市的ETL数据处理
大数据·数据库·数据仓库·数据挖掘·数据分析·etl·推荐算法
桂成林2 天前
Hive UDF 开发实战:MD5 哈希函数实现
hive·hadoop·哈希算法