Hive中常出现的错误(不定时更新)

1.加载数据失败

html 复制代码
hive> load data local inpath '/home/user/hive.txt' into table studentl
    > ;
FAILED: SemanticException [Error 10001]: Line 1:56 Table not found 'studentl'
hive> load data local inpath '/home/user/hive.txt' into table student;
Loading data to table default.student
Failed with exception Unable to move source file:/home/user/hive.txt to destination hdfs://com.qiyu02:8020/user/hive/warehouse/student/hive.txt
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask

在hive日志发现

html 复制代码
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/hive/warehouse/student/hive.txt could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and no node(s) are excluded in this operation.
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1503)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3124)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:636)
	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:188)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:476)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

	at org.apache.hadoop.ipc.Client.call(Client.java:1411)
	at org.apache.hadoop.ipc.Client.call(Client.java:1364)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
	at com.sun.proxy.$Proxy19.addBlock(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:391)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccesThere are 1 datanode(s) running and no node(s) are excluded in this operation.
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1503)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3124)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:636)
	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:188)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:476)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

	at org.apache.hadoop.ipc.Client.call(Client.java:1411)
	at org.apache.hadoop.ipc.Client.call(Client.java:1364)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
	at com.sun.proxy.$Proxy19.addBlock(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:391)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAcces
html 复制代码
<span style="color:#ff0000">There are 1 datanode(s) running and no node(s) are excluded in this operation.</span>

大概是datanode(s) 挂掉了 ,敲下jps验证一下,果真如此,不得不重启一下hadoop进程了

2.Hive中运行Mapredurce程序时:

错误:

Ended Job = job_1504575362948_0022 with errors Error during job, obtaining debugging information...

html 复制代码
hive (person)> select password from user1;
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1504575362948_0022, Tracking URL = http://com.qiyu02:8088/proxy/application_1504575362948_0022/
Kill Command = /opt/modules/hadoop-2.5.0-cdh5.3.6/bin/hadoop job  -kill job_1504575362948_0022
Hadoop job information for Stage-1: number of mappers: 0; number of reducers: 0
2017-09-05 11:44:53,804 Stage-1 map = 0%,  reduce = 0%
Ended Job = job_1504575362948_0022 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched: 
Stage-Stage-1:  HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

网上说法一大堆!说什么内存不够啊、ResourceManger没成功启动啊....等等全试了一遍,还是没解决。。。

后来猜测可能是内核的原因,改下内核,重新启动就好了

3. hive on tez ,yarn 任务一直都running 状态。

修改 tez :默认 tez.session.am.dag.submit.timeout.secs=600

改成 0

修改 hive : tez.session.am.dag.submit.timeout.secs = xxxx

改成 0

资源有限的话 改成 0,越早释放资源越好

相关推荐
北随琛烬入27 分钟前
Spark(10)配置Hadoop集群-集群配置
java·hadoop·spark
蒋星熠3 小时前
在VMware下Hadoop分布式集群环境的配置--基于Yarn模式的一个Master节点、两个Slaver(Worker)节点的配置
大数据·linux·hadoop·分布式·ubuntu·docker
啊迷诺斯1 天前
hadoop的常用命令
hadoop
Loving_enjoy1 天前
基于Hadoop的明星社交媒体影响力数据挖掘平台:设计与实现
大数据·hadoop·数据挖掘
weixin_307779131 天前
使用C#实现从Hive的CREATE TABLE语句中提取分区字段名和数据类型
开发语言·数据仓库·hive·c#
一个天蝎座 白勺 程序猿1 天前
大数据(4.6)Hive执行引擎选型终极指南:MapReduce/Tez/Spark性能实测×万亿级数据资源配置公式
大数据·hive·mapreduce
一个天蝎座 白勺 程序猿2 天前
大数据(4.5)Hive聚合函数深度解析:从基础统计到多维聚合的12个生产级技巧
大数据·hive·hadoop
浩浩kids2 天前
Hadoop•踩过的SHIT
大数据·hadoop·分布式
weixin_307779132 天前
C#实现HiveQL建表语句中特殊数据类型的包裹
开发语言·数据仓库·hive·c#
一个天蝎座 白勺 程序猿3 天前
大数据(4.2)Hive核心操作实战指南:表创建、数据加载与分区/分桶设计深度解析
大数据·hive·hadoop