Hive 本地启动时报错 Persistence Manager has been closed

Hive 本地启动时报错 Persistence Manager has been closed

cobol 复制代码
2024-09-07 17:21:45 ERROR RetryingHMSHandler:215 - Retrying HMSHandler after 2000 ms (attempt 2 of 10) with error: javax.jdo.JDOFatalUserException: Persistence Manager has been closed
	at org.datanucleus.api.jdo.JDOPersistenceManager.assertIsOpen(JDOPersistenceManager.java:2212)
	at org.datanucleus.api.jdo.JDOPersistenceManager.evictAll(JDOPersistenceManager.java:494)
	at org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:804)
	at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.close(ObjectStore.java:3740)
	at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3633)
	at org.apache.hadoop.hive.metastore.ObjectStore.getDatabaseInternal(ObjectStore.java:1019)
	at org.apache.hadoop.hive.metastore.ObjectStore.getDatabase(ObjectStore.java:991)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
	at com.sun.proxy.$Proxy118.getDatabase(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database_core(HiveMetaStore.java:1370)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database(HiveMetaStore.java:1352)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
	at com.sun.proxy.$Proxy120.get_database(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1511)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1506)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
	at com.sun.proxy.$Proxy122.getDatabase(Unknown Source)
	at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1662)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:2166)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:2160)
	at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeShowTables(DDLSemanticAnalyzer.java:2641)
	at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:389)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:659)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1826)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1773)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1768)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
	at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:197)
	at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:260)
	at org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:541)
	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:527)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
	at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
	at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
	at com.sun.proxy.$Proxy128.executeStatementAsync(Unknown Source)
	at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:312)
	at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:562)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

过程

当在本地多个线程启动 HiveMetaStore 时,会根据 conf 创建和使用各自线程的 ObjectStore。

ObjectStore 创建时会设置 HiveConf,并且在设置 HiveConf 时,会判断 HiveConf 是否进行了更新,这时如果没有设置 forTwoMetastoreTesting 参数的话,会关闭 ObjectStore类静态对象 pmf (PersistenceManagerFactory),而 pmf 关闭时也会将内部缓存的JOPPersistenceManager 全部关闭。


这就出现了个问题,ObjectStore 内部缓存着静态变量 PersistenceManagerFactory pmf 创建的 PersistenceManager pm,如果有多个 ObjectStore,其中一个 ObjectStore 关闭了 pmf,pmf 关闭了所有内部所有的 PersistenceManager,那么其他的 ObjectStore 缓存的 pm 肯定是关闭的,所以这个时候就出现了 Persistence Manager has been closed 的问题。

概括

当在本地多个线程启动 HiveMetaStore 时,会根据 conf 创建和使用各自线程的 ObjectStore。

ObjectStore 创建时会设置 HiveConf,并且在设置 HiveConf 时,会判断 HiveConf 是否进行了更新,这时如果没有设置 forTwoMetastoreTesting 参数的话,会关闭 ObjectStore类静态对象 pmf (PersistenceManagerFactory),而 pmf 关闭时也会将内部缓存的JOPPersistenceManager 全部关闭。

这就出现了个问题,ObjectStore 内部缓存着静态变量 PersistenceManagerFactory pmf 创建的 PersistenceManager pm,如果有多个 ObjectStore,其中一个 ObjectStore 关闭了 pmf,pmf 关闭了所有内部所有的 PersistenceManager,那么其他的 ObjectStore 缓存的 pm 肯定是关闭的,所以这个时候就出现了 Persistence Manager has been closed 的问题。

解决办法:

  1. 设置 ObjectStore 类中静态变量 forTwoMetastoreTesting 为 true (推荐)

  2. 使用单个线程启动 HiveMetaStore

  3. 修改源码使多个线程启动的 HiveMetaStore 共享同一个ObjectStore

相关推荐
计艺回忆路1 小时前
Hive自定义函数(UDF)开发和应用流程
hive·自定义函数·udf
万能小锦鲤15 小时前
《大数据技术原理与应用》实验报告三 熟悉HBase常用操作
java·hadoop·eclipse·hbase·shell·vmware·实验报告
王小王-1231 天前
基于Hadoop与LightFM的美妆推荐系统设计与实现
大数据·hive·hadoop·大数据美妆推荐系统·美妆商品用户行为·美妆电商
一切顺势而行1 天前
hadoop 集群问题处理
大数据·hadoop·分布式
万能小锦鲤2 天前
《大数据技术原理与应用》实验报告七 熟悉 Spark 初级编程实践
hive·hadoop·ubuntu·flink·spark·vmware·实验报告
项目題供诗2 天前
Hadoop(二)
大数据·hadoop·分布式
万能小锦鲤2 天前
《大数据技术原理与应用》实验报告五 熟悉 Hive 的基本操作
hive·hadoop·ubuntu·eclipse·vmware·实验报告·hiveql
張萠飛2 天前
flink sql如何对hive string类型的时间戳进行排序
hive·sql·flink
張萠飛2 天前
flink sql读hive catalog数据,将string类型的时间戳数据排序后写入kafka,如何保障写入kafka的数据是有序的
hive·sql·flink
随心............3 天前
hive的相关的优化
数据仓库·hive·hadoop