Hive 本地启动时报错 Persistence Manager has been closed

Hive 本地启动时报错 Persistence Manager has been closed

cobol 复制代码
2024-09-07 17:21:45 ERROR RetryingHMSHandler:215 - Retrying HMSHandler after 2000 ms (attempt 2 of 10) with error: javax.jdo.JDOFatalUserException: Persistence Manager has been closed
	at org.datanucleus.api.jdo.JDOPersistenceManager.assertIsOpen(JDOPersistenceManager.java:2212)
	at org.datanucleus.api.jdo.JDOPersistenceManager.evictAll(JDOPersistenceManager.java:494)
	at org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:804)
	at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.close(ObjectStore.java:3740)
	at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3633)
	at org.apache.hadoop.hive.metastore.ObjectStore.getDatabaseInternal(ObjectStore.java:1019)
	at org.apache.hadoop.hive.metastore.ObjectStore.getDatabase(ObjectStore.java:991)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
	at com.sun.proxy.$Proxy118.getDatabase(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database_core(HiveMetaStore.java:1370)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database(HiveMetaStore.java:1352)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
	at com.sun.proxy.$Proxy120.get_database(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1511)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1506)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
	at com.sun.proxy.$Proxy122.getDatabase(Unknown Source)
	at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1662)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:2166)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:2160)
	at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeShowTables(DDLSemanticAnalyzer.java:2641)
	at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:389)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:659)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1826)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1773)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1768)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
	at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:197)
	at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:260)
	at org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:541)
	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:527)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
	at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
	at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
	at com.sun.proxy.$Proxy128.executeStatementAsync(Unknown Source)
	at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:312)
	at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:562)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

过程

当在本地多个线程启动 HiveMetaStore 时,会根据 conf 创建和使用各自线程的 ObjectStore。

ObjectStore 创建时会设置 HiveConf,并且在设置 HiveConf 时,会判断 HiveConf 是否进行了更新,这时如果没有设置 forTwoMetastoreTesting 参数的话,会关闭 ObjectStore类静态对象 pmf (PersistenceManagerFactory),而 pmf 关闭时也会将内部缓存的JOPPersistenceManager 全部关闭。


这就出现了个问题,ObjectStore 内部缓存着静态变量 PersistenceManagerFactory pmf 创建的 PersistenceManager pm,如果有多个 ObjectStore,其中一个 ObjectStore 关闭了 pmf,pmf 关闭了所有内部所有的 PersistenceManager,那么其他的 ObjectStore 缓存的 pm 肯定是关闭的,所以这个时候就出现了 Persistence Manager has been closed 的问题。

概括

当在本地多个线程启动 HiveMetaStore 时,会根据 conf 创建和使用各自线程的 ObjectStore。

ObjectStore 创建时会设置 HiveConf,并且在设置 HiveConf 时,会判断 HiveConf 是否进行了更新,这时如果没有设置 forTwoMetastoreTesting 参数的话,会关闭 ObjectStore类静态对象 pmf (PersistenceManagerFactory),而 pmf 关闭时也会将内部缓存的JOPPersistenceManager 全部关闭。

这就出现了个问题,ObjectStore 内部缓存着静态变量 PersistenceManagerFactory pmf 创建的 PersistenceManager pm,如果有多个 ObjectStore,其中一个 ObjectStore 关闭了 pmf,pmf 关闭了所有内部所有的 PersistenceManager,那么其他的 ObjectStore 缓存的 pm 肯定是关闭的,所以这个时候就出现了 Persistence Manager has been closed 的问题。

解决办法:

  1. 设置 ObjectStore 类中静态变量 forTwoMetastoreTesting 为 true (推荐)

  2. 使用单个线程启动 HiveMetaStore

  3. 修改源码使多个线程启动的 HiveMetaStore 共享同一个ObjectStore

相关推荐
IT学长编程1 小时前
计算机毕业设计 基于EChants的海洋气象数据可视化平台设计与实现 Python 大数据毕业设计 Hadoop毕业设计选题【附源码+文档报告+安装调试】
大数据·hadoop·python·毕业设计·课程设计·毕业论文·海洋气象数据可视化平台
Yana_Zeng5 小时前
win10安装spark3.1详细流程(小白用)
hadoop·windows·spark
笨蛋少年派8 小时前
HDFS简介
大数据·hadoop·hdfs
工作中的程序员1 天前
hive sql优化基础
hive·sql
K_i1342 天前
中国电信用户行为实时分析系统运维实战
hadoop·mysql
风跟我说过她2 天前
Sqoop的安装与配置
hive·hadoop·经验分享·centos·hbase·sqoop
K_i1342 天前
电信大数据实战:MySQL与Hadoop高效同步
大数据·hadoop·mysql
weixin_525936333 天前
金融大数据处理与分析
hadoop·python·hdfs·金融·数据分析·spark·matplotlib
秃头菜狗3 天前
十三、格式化 HDFS 文件系统、启动伪分布式集群
大数据·hadoop·hdfs
笨蛋少年派3 天前
Hadoop简介
大数据·hadoop