Hive 本地启动时报错 Persistence Manager has been closed

Hive 本地启动时报错 Persistence Manager has been closed

cobol 复制代码
2024-09-07 17:21:45 ERROR RetryingHMSHandler:215 - Retrying HMSHandler after 2000 ms (attempt 2 of 10) with error: javax.jdo.JDOFatalUserException: Persistence Manager has been closed
	at org.datanucleus.api.jdo.JDOPersistenceManager.assertIsOpen(JDOPersistenceManager.java:2212)
	at org.datanucleus.api.jdo.JDOPersistenceManager.evictAll(JDOPersistenceManager.java:494)
	at org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:804)
	at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.close(ObjectStore.java:3740)
	at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3633)
	at org.apache.hadoop.hive.metastore.ObjectStore.getDatabaseInternal(ObjectStore.java:1019)
	at org.apache.hadoop.hive.metastore.ObjectStore.getDatabase(ObjectStore.java:991)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
	at com.sun.proxy.$Proxy118.getDatabase(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database_core(HiveMetaStore.java:1370)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database(HiveMetaStore.java:1352)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
	at com.sun.proxy.$Proxy120.get_database(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1511)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1506)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
	at com.sun.proxy.$Proxy122.getDatabase(Unknown Source)
	at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1662)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:2166)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:2160)
	at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeShowTables(DDLSemanticAnalyzer.java:2641)
	at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:389)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:659)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1826)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1773)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1768)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
	at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:197)
	at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:260)
	at org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:541)
	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:527)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
	at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
	at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
	at com.sun.proxy.$Proxy128.executeStatementAsync(Unknown Source)
	at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:312)
	at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:562)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

过程

当在本地多个线程启动 HiveMetaStore 时,会根据 conf 创建和使用各自线程的 ObjectStore。

ObjectStore 创建时会设置 HiveConf,并且在设置 HiveConf 时,会判断 HiveConf 是否进行了更新,这时如果没有设置 forTwoMetastoreTesting 参数的话,会关闭 ObjectStore类静态对象 pmf (PersistenceManagerFactory),而 pmf 关闭时也会将内部缓存的JOPPersistenceManager 全部关闭。


这就出现了个问题,ObjectStore 内部缓存着静态变量 PersistenceManagerFactory pmf 创建的 PersistenceManager pm,如果有多个 ObjectStore,其中一个 ObjectStore 关闭了 pmf,pmf 关闭了所有内部所有的 PersistenceManager,那么其他的 ObjectStore 缓存的 pm 肯定是关闭的,所以这个时候就出现了 Persistence Manager has been closed 的问题。

概括

当在本地多个线程启动 HiveMetaStore 时,会根据 conf 创建和使用各自线程的 ObjectStore。

ObjectStore 创建时会设置 HiveConf,并且在设置 HiveConf 时,会判断 HiveConf 是否进行了更新,这时如果没有设置 forTwoMetastoreTesting 参数的话,会关闭 ObjectStore类静态对象 pmf (PersistenceManagerFactory),而 pmf 关闭时也会将内部缓存的JOPPersistenceManager 全部关闭。

这就出现了个问题,ObjectStore 内部缓存着静态变量 PersistenceManagerFactory pmf 创建的 PersistenceManager pm,如果有多个 ObjectStore,其中一个 ObjectStore 关闭了 pmf,pmf 关闭了所有内部所有的 PersistenceManager,那么其他的 ObjectStore 缓存的 pm 肯定是关闭的,所以这个时候就出现了 Persistence Manager has been closed 的问题。

解决办法:

  1. 设置 ObjectStore 类中静态变量 forTwoMetastoreTesting 为 true (推荐)

  2. 使用单个线程启动 HiveMetaStore

  3. 修改源码使多个线程启动的 HiveMetaStore 共享同一个ObjectStore

相关推荐
程序猿 董班长1 天前
springboot配置多数据源(mysql、hive)
hive·spring boot·mysql
XueminXu1 天前
Hive使用Tez引擎出现OOM的解决方法
hive·oom·tez·datagrip·container·outofmemory
zzu123zsw2 天前
01-Hadoop简介与生态系统
hadoop
鸿儒之观2 天前
hadoop 框架 jar下载
大数据·hadoop·jar
2302_799525742 天前
【Hadoop】Hadoop集群安装中出现的问题
linux·hadoop
IT研究室2 天前
大数据毕业设计选题推荐-基于大数据的分化型甲状腺癌复发数据可视化分析系统-Spark-Hadoop-Bigdata
大数据·hadoop·信息可视化·spark·毕业设计·源码·bigdata
计算机编程小央姐2 天前
数据安全成焦点:基于Hadoop+Spark的信用卡诈骗分析系统实战教程
大数据·hadoop·python·spark·毕业设计·课程设计·dash
BYSJMG2 天前
大数据毕业设计推荐:基于Spark的零售时尚精品店销售数据分析系统【Hadoop+python+spark】
大数据·hadoop·python·spark·django·课程设计
项目題供诗3 天前
Hadoop(七)
大数据·hadoop·分布式
小蒜学长3 天前
基于Hadoop的网约车公司数据分析系统设计(代码+数据库+LW)
java·大数据·数据库·hadoop·spring boot·后端