【已解决】Atlas 导入 Hive 元数据,执行 import-hive.sh 报错

部署完 Atlas 之后,尝试导入 Hive 元数据,遇到了一些错误,特此记录一下,方便你我他。

执行 import-hive.sh 报错

复制代码
[omc@hadoop102 apache-atlas-2.2.0]$ hook-bin/import-hive.sh
Using Hive configuration directory [/opt/module/hive/conf]
Log file for import is /opt/module/apache-atlas-2.2.0/logs/import-hive.log
2023-12-17T15:48:37,827 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - delete non existing flag : false
2023-12-17T15:48:37,846 INFO [main] org.apache.atlas.ApplicationProperties - Looking for atlas-application.properties in classpath
2023-12-17T15:48:37,847 INFO [main] org.apache.atlas.ApplicationProperties - Loading atlas-application.properties from file:/opt/module/hive/conf/atlas-application.properties
2023-12-17T15:48:37,883 INFO [main] org.apache.atlas.ApplicationProperties - Using graphdb backend 'janus'
2023-12-17T15:48:37,883 INFO [main] org.apache.atlas.ApplicationProperties - Using storage backend 'hbase2'
2023-12-17T15:48:37,883 INFO [main] org.apache.atlas.ApplicationProperties - Using index backend 'solr'
2023-12-17T15:48:37,883 INFO [main] org.apache.atlas.ApplicationProperties - Atlas is running in MODE: PROD.
2023-12-17T15:48:37,887 INFO [main] org.apache.atlas.ApplicationProperties - Setting solr.wait-searcher property 'false'
2023-12-17T15:48:37,887 INFO [main] org.apache.atlas.ApplicationProperties - Setting index.search.map-name property 'false'
2023-12-17T15:48:37,887 INFO [main] org.apache.atlas.ApplicationProperties - Setting atlas.graph.index.search.max-result-set-size = 150
2023-12-17T15:48:37,887 INFO [main] org.apache.atlas.ApplicationProperties - Setting atlas.graph.index.search.solr.wait-searcher = false
2023-12-17T15:48:37,887 INFO [main] org.apache.atlas.ApplicationProperties - Property (set to default) atlas.graph.cache.db-cache = true
2023-12-17T15:48:37,887 INFO [main] org.apache.atlas.ApplicationProperties - Property (set to default) atlas.graph.cache.db-cache-clean-wait = 20
2023-12-17T15:48:37,887 INFO [main] org.apache.atlas.ApplicationProperties - Property (set to default) atlas.graph.cache.db-cache-size = 0.5
2023-12-17T15:48:37,887 INFO [main] org.apache.atlas.ApplicationProperties - Property (set to default) atlas.graph.cache.tx-cache-size = 15000
2023-12-17T15:48:37,887 INFO [main] org.apache.atlas.ApplicationProperties - Property (set to default) atlas.graph.cache.tx-dirty-size = 120
Enter username for atlas :- admin
Enter password for atlas :-
2023-12-17T15:48:44,649 INFO [main] org.apache.atlas.AtlasBaseClient - Client has only one service URL, will use that for all actions: http://192.168.10.102:21000
2023-12-17T15:48:44,690 INFO [main] org.apache.hadoop.hive.conf.HiveConf - Found configuration file file:/opt/module/hive/conf/hive-site.xml
2023-12-17T15:48:45,786 WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-12-17T15:48:45,884 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStoreClient - Trying to connect to metastore with URI thrift://hadoop102:9083
2023-12-17T15:48:45,921 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStoreClient - Opened a connection to metastore, current connections: 1
2023-12-17T15:48:46,013 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStoreClient - Connected to metastore.
2023-12-17T15:48:46,013 INFO [main] org.apache.hadoop.hive.metastore.RetryingMetaStoreClient - RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=omc (auth:SIMPLE) retries=1 delay=1 lifetime=0
2023-12-17T15:48:46,291 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Importing Hive metadata
2023-12-17T15:48:46,310 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Found 7 databases
2023-12-17T15:48:46,439 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/uniqueAttribute/type/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2023-12-17T15:48:46,646 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Database default is already registered - id=eb2c0a75-1984-404c-8cdb-0dcbfc3e74ae. Updating it.
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/HdfsConfiguration
        at org.apache.atlas.utils.HdfsNameServiceResolver.init(HdfsNameServiceResolver.java:139)
        at org.apache.atlas.utils.HdfsNameServiceResolver.<clinit>(HdfsNameServiceResolver.java:47)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.toDbEntity(HiveMetaStoreBridge.java:606)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:461)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:322)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:292)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:192)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.HdfsConfiguration
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 7 more
Failed to import Hive Meta Data!!!
[omc@hadoop102 apache-atlas-2.2.0]$

在 $HIVE_HOME/lib 目录下添加 hadoop-hdfs-client-3.1.3.jar 即可。

说明:版本号要和自己的 Hadoop 版本号一致。

相关推荐
十月南城4 天前
数据湖技术对比——Iceberg、Hudi、Delta的表格格式与维护策略
大数据·数据库·数据仓库·hive·hadoop·spark
王九思4 天前
Hive Thrift Server 介绍
数据仓库·hive·hadoop
Asher05094 天前
Hive核心知识:从基础到实战全解析
数据仓库·hive·hadoop
xhaoDream4 天前
Hive3.1.3 配置 Tez 引擎
大数据·hive·tez
TTBIGDATA6 天前
【Atlas】Atlas 搜索时报 `__AtlasUserProfile` 不存在导致事务回滚
开发语言·python·ambari·kerberos·ranger·atlas·bigtop
一号IT男7 天前
Hive中GROUPING SETS功能详解
数据仓库·hive·hadoop
500佰7 天前
Hive常见故障多案例FAQ宝典 --项目总结(宝典一)
大数据·linux·数据仓库·hive·hadoop·云计算·运维开发
好学且牛逼的马10 天前
从“配置地狱“到“云原生时代“:Spring Boot 1.x到4.x演进全记录与核心知识点详解
hive·spring boot·云原生
Timer_Cooker12 天前
Hive Sum(null)编译报错分析
数据仓库·hive·hadoop