Hive 本地启动时报错 Persistence Manager has been closed

Hive 本地启动时报错 Persistence Manager has been closed

cobol 复制代码
2024-09-07 17:21:45 ERROR RetryingHMSHandler:215 - Retrying HMSHandler after 2000 ms (attempt 2 of 10) with error: javax.jdo.JDOFatalUserException: Persistence Manager has been closed
	at org.datanucleus.api.jdo.JDOPersistenceManager.assertIsOpen(JDOPersistenceManager.java:2212)
	at org.datanucleus.api.jdo.JDOPersistenceManager.evictAll(JDOPersistenceManager.java:494)
	at org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:804)
	at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.close(ObjectStore.java:3740)
	at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3633)
	at org.apache.hadoop.hive.metastore.ObjectStore.getDatabaseInternal(ObjectStore.java:1019)
	at org.apache.hadoop.hive.metastore.ObjectStore.getDatabase(ObjectStore.java:991)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
	at com.sun.proxy.$Proxy118.getDatabase(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database_core(HiveMetaStore.java:1370)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database(HiveMetaStore.java:1352)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
	at com.sun.proxy.$Proxy120.get_database(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1511)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1506)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
	at com.sun.proxy.$Proxy122.getDatabase(Unknown Source)
	at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1662)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:2166)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:2160)
	at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeShowTables(DDLSemanticAnalyzer.java:2641)
	at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:389)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:659)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1826)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1773)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1768)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
	at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:197)
	at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:260)
	at org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:541)
	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:527)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
	at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
	at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
	at com.sun.proxy.$Proxy128.executeStatementAsync(Unknown Source)
	at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:312)
	at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:562)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

过程

当在本地多个线程启动 HiveMetaStore 时,会根据 conf 创建和使用各自线程的 ObjectStore。

ObjectStore 创建时会设置 HiveConf,并且在设置 HiveConf 时,会判断 HiveConf 是否进行了更新,这时如果没有设置 forTwoMetastoreTesting 参数的话,会关闭 ObjectStore类静态对象 pmf (PersistenceManagerFactory),而 pmf 关闭时也会将内部缓存的JOPPersistenceManager 全部关闭。


这就出现了个问题,ObjectStore 内部缓存着静态变量 PersistenceManagerFactory pmf 创建的 PersistenceManager pm,如果有多个 ObjectStore,其中一个 ObjectStore 关闭了 pmf,pmf 关闭了所有内部所有的 PersistenceManager,那么其他的 ObjectStore 缓存的 pm 肯定是关闭的,所以这个时候就出现了 Persistence Manager has been closed 的问题。

概括

当在本地多个线程启动 HiveMetaStore 时,会根据 conf 创建和使用各自线程的 ObjectStore。

ObjectStore 创建时会设置 HiveConf,并且在设置 HiveConf 时,会判断 HiveConf 是否进行了更新,这时如果没有设置 forTwoMetastoreTesting 参数的话,会关闭 ObjectStore类静态对象 pmf (PersistenceManagerFactory),而 pmf 关闭时也会将内部缓存的JOPPersistenceManager 全部关闭。

这就出现了个问题,ObjectStore 内部缓存着静态变量 PersistenceManagerFactory pmf 创建的 PersistenceManager pm,如果有多个 ObjectStore,其中一个 ObjectStore 关闭了 pmf,pmf 关闭了所有内部所有的 PersistenceManager,那么其他的 ObjectStore 缓存的 pm 肯定是关闭的,所以这个时候就出现了 Persistence Manager has been closed 的问题。

解决办法:

  1. 设置 ObjectStore 类中静态变量 forTwoMetastoreTesting 为 true (推荐)

  2. 使用单个线程启动 HiveMetaStore

  3. 修改源码使多个线程启动的 HiveMetaStore 共享同一个ObjectStore

相关推荐
lzhlizihang1 小时前
【Hive sql 面试题】求出各类型专利top 10申请人,以及对应的专利申请数(难)
大数据·hive·sql·面试题
Hsu_kk1 小时前
Hive 查询各类型专利 Top 10 申请人及对应的专利申请数
数据仓库·hive·hadoop
静听山水1 小时前
Hive 的数据存储单元结构
hive
大数据编程之光1 小时前
Hive 查询各类型专利 top10 申请人及专利申请数
大数据·数据仓库·hive·hadoop
杰克逊的日记1 小时前
Hive详解
数据仓库·hive·hadoop
上辈子杀猪这辈子学IT1 小时前
【Zookeeper集群搭建】安装zookeeper、zookeeper集群配置、zookeeper启动与关闭、zookeeper的shell命令操作
linux·hadoop·zookeeper·centos·debian
Hsu_kk3 小时前
Hive 查询用户连续三天登录的所有记录
数据仓库·hive·hadoop
kakwooi7 小时前
Hadoop---MapReduce(3)
大数据·hadoop·mapreduce
windy1a7 小时前
【c知道】Hadoop工作原理。
hadoop
油头少年_w13 小时前
大数据导论及分布式存储HadoopHDFS入门
大数据·hadoop·hdfs