Hive 本地启动时报错 Persistence Manager has been closed

Hive 本地启动时报错 Persistence Manager has been closed

cobol 复制代码
2024-09-07 17:21:45 ERROR RetryingHMSHandler:215 - Retrying HMSHandler after 2000 ms (attempt 2 of 10) with error: javax.jdo.JDOFatalUserException: Persistence Manager has been closed
	at org.datanucleus.api.jdo.JDOPersistenceManager.assertIsOpen(JDOPersistenceManager.java:2212)
	at org.datanucleus.api.jdo.JDOPersistenceManager.evictAll(JDOPersistenceManager.java:494)
	at org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:804)
	at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.close(ObjectStore.java:3740)
	at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3633)
	at org.apache.hadoop.hive.metastore.ObjectStore.getDatabaseInternal(ObjectStore.java:1019)
	at org.apache.hadoop.hive.metastore.ObjectStore.getDatabase(ObjectStore.java:991)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
	at com.sun.proxy.$Proxy118.getDatabase(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database_core(HiveMetaStore.java:1370)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database(HiveMetaStore.java:1352)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
	at com.sun.proxy.$Proxy120.get_database(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1511)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1506)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
	at com.sun.proxy.$Proxy122.getDatabase(Unknown Source)
	at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1662)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:2166)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:2160)
	at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeShowTables(DDLSemanticAnalyzer.java:2641)
	at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:389)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:659)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1826)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1773)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1768)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
	at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:197)
	at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:260)
	at org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:541)
	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:527)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
	at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
	at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
	at com.sun.proxy.$Proxy128.executeStatementAsync(Unknown Source)
	at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:312)
	at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:562)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

过程

当在本地多个线程启动 HiveMetaStore 时,会根据 conf 创建和使用各自线程的 ObjectStore。

ObjectStore 创建时会设置 HiveConf,并且在设置 HiveConf 时,会判断 HiveConf 是否进行了更新,这时如果没有设置 forTwoMetastoreTesting 参数的话,会关闭 ObjectStore类静态对象 pmf (PersistenceManagerFactory),而 pmf 关闭时也会将内部缓存的JOPPersistenceManager 全部关闭。


这就出现了个问题,ObjectStore 内部缓存着静态变量 PersistenceManagerFactory pmf 创建的 PersistenceManager pm,如果有多个 ObjectStore,其中一个 ObjectStore 关闭了 pmf,pmf 关闭了所有内部所有的 PersistenceManager,那么其他的 ObjectStore 缓存的 pm 肯定是关闭的,所以这个时候就出现了 Persistence Manager has been closed 的问题。

概括

当在本地多个线程启动 HiveMetaStore 时,会根据 conf 创建和使用各自线程的 ObjectStore。

ObjectStore 创建时会设置 HiveConf,并且在设置 HiveConf 时,会判断 HiveConf 是否进行了更新,这时如果没有设置 forTwoMetastoreTesting 参数的话,会关闭 ObjectStore类静态对象 pmf (PersistenceManagerFactory),而 pmf 关闭时也会将内部缓存的JOPPersistenceManager 全部关闭。

这就出现了个问题,ObjectStore 内部缓存着静态变量 PersistenceManagerFactory pmf 创建的 PersistenceManager pm,如果有多个 ObjectStore,其中一个 ObjectStore 关闭了 pmf,pmf 关闭了所有内部所有的 PersistenceManager,那么其他的 ObjectStore 缓存的 pm 肯定是关闭的,所以这个时候就出现了 Persistence Manager has been closed 的问题。

解决办法:

  1. 设置 ObjectStore 类中静态变量 forTwoMetastoreTesting 为 true (推荐)

  2. 使用单个线程启动 HiveMetaStore

  3. 修改源码使多个线程启动的 HiveMetaStore 共享同一个ObjectStore

相关推荐
JessieZeng aaa10 小时前
CSV文件数据导入hive
数据仓库·hive·hadoop
Yz987616 小时前
hive复杂数据类型Array & Map & Struct & 炸裂函数explode
大数据·数据库·数据仓库·hive·hadoop·数据库开发·big data
EDG Zmjjkk18 小时前
Hive 函数(实例操作版2)
数据仓库·hive·hadoop
那一抹阳光多灿烂1 天前
Spark核心组件解析:Executor、RDD与缓存优化
hadoop·spark
Yz98761 天前
Hive分桶超详细!!!
大数据·数据仓库·hive·hadoop·hdfs·数据库开发·big data
Francek Chen1 天前
【大数据技术基础 | 实验十一】Hive实验:新建Hive表
大数据·数据仓库·hive·hadoop·分布式
出发行进1 天前
Flink错误:一historyserver无法启动,二存在的文件会报错没有那个文件或目录
大数据·linux·hadoop·flink·虚拟机
闲人编程1 天前
Hadoop 使用过程中 15 个常见问题的详细描述、解决方案
大数据·hadoop·eclipse·解决方案·yarn·配置文件
橘子海全栈攻城狮2 天前
【源码+文档+调试讲解】基于Hadoop实现的豆瓣电子图书推荐系统的设计与实现
大数据·数据库·hadoop·spring boot·分布式·后端
二进制_博客2 天前
hadoop集群搭建
java·大数据·hadoop