CDH 6.3.2集成Hudi异常org.codehaus.jackson不存在及开源JDK版本异常

qq_214803292024-02-19 13:18

CDH 6.3.2集成Hudi异常，首先获取hudi源码，地址：git clone https://github.com/apache/hudi.git，进入根目录hudi编译相关jar时，存在2个问题jar包依赖为导入和开源JDK版本问题。异常分别如下所示。

1.编译命令

到hudi根目录编译即可，命令如下所示：

复制代码 
-Dspark3.4  指定spark版本3.4
-Dflink1.18  指定flink版本1.18
-Dhadoop.version=3.3.5 指定hadoop版本3.3.5
mvn clean package -DskipTests -Dcheckstyle.skip -Dspark3.4 -Dflink1.18 -Dscala-2.12 -Dhadoop.version=3.3.5 -Pflink-bundle-shade-hive3

2.jar包依赖问题

复制代码 
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ hudi-common ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 543 source files to /Users/carter/Desktop/study/hudi/hudi-common/target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /Users/carter/Desktop/study/hudi/hudi-common/src/main/java/org/apache/hudi/avro/JsonEncoder.java:[30,28] 程序包org.codehaus.jackson不存在
[ERROR] /Users/carter/Desktop/study/hudi/hudi-common/src/main/java/org/apache/hudi/avro/JsonEncoder.java:[31,28] 程序包org.codehaus.jackson不存在
[ERROR] /Users/carter/Desktop/study/hudi/hudi-common/src/main/java/org/apache/hudi/avro/JsonEncoder.java:[32,28] 程序包org.codehaus.jackson不存在
[ERROR] /Users/carter/Desktop/study/hudi/hudi-common/src/main/java/org/apache/hudi/avro/JsonEncoder.java:[33,33] 程序包org.codehaus.jackson.util不存在
[ERROR] /Users/carter/Desktop/study/hudi/hudi-common/src/main/java/org/apache/hudi/avro/JsonEncoder.java:[63,11] 找不到符号
  符号:   类 JsonGenerator
  位置: 类 org.apache.hudi.avro.JsonEncoder
[ERROR] /Users/carter/Desktop/study/hudi/hudi-common/src/main/java/org/apache/hudi/avro/JsonEncoder.java:[83,26] 找不到符号
  符号:   类 JsonGenerator
  位置: 类 org.apache.hudi.avro.JsonEncoder
[ERROR] /Users/carter/Desktop/study/hudi/hudi-common/src/main/java/org/apache/hudi/avro/JsonEncoder.java:[105,18] 找不到符号
  符号:   类 JsonGenerator
  位置: 类 org.apache.hudi.avro.JsonEncoder
[ERROR] /Users/carter/Desktop/study/hudi/hudi-common/src/main/java/org/apache/hudi/avro/JsonEncoder.java:[179,33] 找不到符号
  符号:   类 JsonGenerator
  位置: 类 org.apache.hudi.avro.JsonEncoder
[INFO] 8 errors 
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Hudi 1.0.0-SNAPSHOT:
[INFO] 
[INFO] Hudi ............................................... SUCCESS [  1.715 s]
[INFO] hudi-tests-common .................................. SUCCESS [  2.741 s]
[INFO] hudi-io ............................................ SUCCESS [  4.245 s]
[INFO] hudi-hadoop-common ................................. SUCCESS [  2.177 s]
[INFO] hudi-common ........................................ FAILURE [  6.635 s]
[INFO] hudi-hadoop-mr ..................................... SKIPPED
[INFO] hudi-sync-common ................................... SKIPPED
[INFO] hudi-hive-sync ..................................... SKIPPED
[INFO] hudi-aws ........................................... SKIPPED
[INFO] hudi-timeline-service .............................. SKIPPED
[INFO] hudi-client ........................................ SKIPPED
[INFO] hudi-client-common ................................. SKIPPED
[INFO] hudi-spark-client .................................. SKIPPED
[INFO] hudi-spark-datasource .............................. SKIPPED
[INFO] hudi-spark-common_2.12 ............................. SKIPPED
[INFO] hudi-spark3-common ................................. SKIPPED
[INFO] hudi-spark3.2plus-common ........................... SKIPPED
[INFO] hudi-spark3.4.x_2.12 ............................... SKIPPED
[INFO] hudi-java-client ................................... SKIPPED
[INFO] hudi-spark_2.12 .................................... SKIPPED
[INFO] hudi-gcp ........................................... SKIPPED
[INFO] hudi-utilities_2.12 ................................ SKIPPED
[INFO] hudi-utilities-bundle_2.12 ......................... SKIPPED
[INFO] hudi-cli ........................................... SKIPPED
[INFO] hudi-flink-client .................................. SKIPPED
[INFO] hudi-datahub-sync .................................. SKIPPED
[INFO] hudi-adb-sync ...................................... SKIPPED
[INFO] hudi-sync .......................................... SKIPPED
[INFO] hudi-hadoop-mr-bundle .............................. SKIPPED
[INFO] hudi-datahub-sync-bundle ........................... SKIPPED
[INFO] hudi-hive-sync-bundle .............................. SKIPPED
[INFO] hudi-aws-bundle .................................... SKIPPED
[INFO] hudi-gcp-bundle .................................... SKIPPED
[INFO] hudi-spark3.4-bundle_2.12 .......................... SKIPPED
[INFO] hudi-presto-bundle ................................. SKIPPED
[INFO] hudi-utilities-slim-bundle_2.12 .................... SKIPPED
[INFO] hudi-timeline-server-bundle ........................ SKIPPED
[INFO] hudi-trino-bundle .................................. SKIPPED
[INFO] hudi-examples ...................................... SKIPPED
[INFO] hudi-examples-common ............................... SKIPPED
[INFO] hudi-examples-spark ................................ SKIPPED
[INFO] hudi-flink-datasource .............................. SKIPPED
[INFO] hudi-flink1.18.x ................................... SKIPPED
[INFO] hudi-flink ......................................... SKIPPED
[INFO] hudi-examples-flink ................................ SKIPPED
[INFO] hudi-examples-java ................................. SKIPPED
[INFO] hudi-flink1.14.x ................................... SKIPPED
[INFO] hudi-flink1.15.x ................................... SKIPPED
[INFO] hudi-flink1.16.x ................................... SKIPPED
[INFO] hudi-flink1.17.x ................................... SKIPPED
[INFO] hudi-kafka-connect ................................. SKIPPED
[INFO] hudi-flink1.18-bundle .............................. SKIPPED
[INFO] hudi-kafka-connect-bundle .......................... SKIPPED
[INFO] hudi-cli-bundle_2.12 ............................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  18.704 s
[INFO] Finished at: 2024-02-18T14:24:57+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project hudi-common: Compilation failure: Compilation failure: 
[ERROR] /Users/carter/Desktop/study/hudi/hudi-common/src/main/java/org/apache/hudi/avro/JsonEncoder.java:[30,28] 程序包org.codehaus.jackson不存在
[ERROR] /Users/carter/Desktop/study/hudi/hudi-common/src/main/java/org/apache/hudi/avro/JsonEncoder.java:[31,28] 程序包org.codehaus.jackson不存在
[ERROR] /Users/carter/Desktop/study/hudi/hudi-common/src/main/java/org/apache/hudi/avro/JsonEncoder.java:[32,28] 程序包org.codehaus.jackson不存在
[ERROR] /Users/carter/Desktop/study/hudi/hudi-common/src/main/java/org/apache/hudi/avro/JsonEncoder.java:[33,33] 程序包org.codehaus.jackson.util不存在
[ERROR] /Users/carter/Desktop/study/hudi/hudi-common/src/main/java/org/apache/hudi/avro/JsonEncoder.java:[63,11] 找不到符号
[ERROR]   符号:   类 JsonGenerator
[ERROR]   位置: 类 org.apache.hudi.avro.JsonEncoder
[ERROR] /Users/carter/Desktop/study/hudi/hudi-common/src/main/java/org/apache/hudi/avro/JsonEncoder.java:[83,26] 找不到符号
[ERROR]   符号:   类 JsonGenerator
[ERROR]   位置: 类 org.apache.hudi.avro.JsonEncoder
[ERROR] /Users/carter/Desktop/study/hudi/hudi-common/src/main/java/org/apache/hudi/avro/JsonEncoder.java:[105,18] 找不到符号
[ERROR]   符号:   类 JsonGenerator
[ERROR]   位置: 类 org.apache.hudi.avro.JsonEncoder
[ERROR] /Users/carter/Desktop/study/hudi/hudi-common/src/main/java/org/apache/hudi/avro/JsonEncoder.java:[179,33] 找不到符号
[ERROR]   符号:   类 JsonGenerator
[ERROR]   位置: 类 org.apache.hudi.avro.JsonEncoder
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :hudi-common
(base) carter@MacBook-Pro hudi %

报错如下图所示：

解决方案：

复制代码 
    <dependency>
      <groupId>org.codehaus.jackson</groupId>
      <artifactId>jackson-core-asl</artifactId>
      <version>1.9.13</version>
    </dependency>

3.开源jdk问题

此问题必须使用官方jdk，即便1.8版本开源JDK也不行。

报错信息：

复制代码 
[INFO] hudi-cli-bundle_2.12 ............................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  30.773 s
[INFO] Finished at: 2024-02-18T10:20:11+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project hudi-common: Compilation failure: Compilation failure: 
[ERROR] 无法访问org.apache.hadoop.shaded.org.apache.avro.reflect.Stringable
[ERROR]   找不到org.apache.hadoop.shaded.org.apache.avro.reflect.Stringable的类文件
[ERROR] /Users/carter/Desktop/study/hudi/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:[282,9] 对于collect(java.util.stream.Collector<org.apache.hudi.common.model.HoodieColumnRangeMetadata<java.lang.Comparable>,capture#1, 共 ?,java.util.Map<java.lang.String,org.apache.hudi.common.model.HoodieColumnRangeMetadata<java.lang.Comparable>>>), 找不到合适的方法
[ERROR]     方法 java.util.stream.Stream.<R>collect(java.util.function.Supplier<R>,java.util.function.BiConsumer<R,? super org.apache.hudi.common.model.HoodieColumnRangeMetadata>,java.util.function.BiConsumer<R,R>)不适用
[ERROR]       (无法推断类型变量 R
[ERROR]         (实际参数列表和形式参数列表长度不同))
[ERROR]     方法 java.util.stream.Stream.<R,A>collect(java.util.stream.Collector<? super org.apache.hudi.common.model.HoodieColumnRangeMetadata,A,R>)不适用
[ERROR]       (无法推断类型变量 R,A
[ERROR]         (参数不匹配; java.util.stream.Collector<org.apache.hudi.common.model.HoodieColumnRangeMetadata<java.lang.Comparable>,capture#1, 共 ?,java.util.Map<java.lang.String,org.apache.hudi.common.model.HoodieColumnRangeMetadata<java.lang.Comparable>>>无法转换为java.util.stream.Collector<? super org.apache.hudi.common.model.HoodieColumnRangeMetadata,A,R>))
[ERROR] /Users/carter/Desktop/study/hudi/hudi-common/src/main/java/org/apache/hudi/common/util/ParquetUtils.java:[369,11] 对于collect(java.util.stream.Collector<org.apache.hudi.common.model.HoodieColumnRangeMetadata<java.lang.Comparable>,capture#2, 共 ?,java.util.Map<java.lang.String,java.util.List<org.apache.hudi.common.model.HoodieColumnRangeMetadata<java.lang.Comparable>>>>), 找不到合适的方法
[ERROR]     方法 java.util.stream.Stream.<R>collect(java.util.function.Supplier<R>,java.util.function.BiConsumer<R,? super org.apache.hudi.common.model.HoodieColumnRangeMetadata>,java.util.function.BiConsumer<R,R>)不适用
[ERROR]       (无法推断类型变量 R
[ERROR]         (实际参数列表和形式参数列表长度不同))
[ERROR]     方法 java.util.stream.Stream.<R,A>collect(java.util.stream.Collector<? super org.apache.hudi.common.model.HoodieColumnRangeMetadata,A,R>)不适用
[ERROR]       (无法推断类型变量 R,A
[ERROR]         (参数不匹配; java.util.stream.Collector<org.apache.hudi.common.model.HoodieColumnRangeMetadata<java.lang.Comparable>,capture#2, 共 ?,java.util.Map<java.lang.String,java.util.List<org.apache.hudi.common.model.HoodieColumnRangeMetadata<java.lang.Comparable>>>>无法转换为java.util.stream.Collector<? super org.apache.hudi.common.model.HoodieColumnRangeMetadata,A,R>))
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :hudi-common
(base) carter@MacBook-Pro hudi %

报错异常截图如下：

解决方法：使用官方JDK版本，不能使用开源版本，只要java -version版本包含openJdk都不可行。

4.相关大数据学习demo地址:

https://github.com/carteryh/big-data

相关推荐
你觉得20510 小时前
哈尔滨工业大学DeepSeek公开课：探索大模型原理、技术与应用从GPT到DeepSeek｜附视频与讲义下载方法
大数据·人工智能·python·gpt·学习·机器学习·aigc
啊喜拔牙10 小时前
1. hadoop 集群的常用命令
java·大数据·开发语言·python·scala
别惊鹊10 小时前
MapReduce工作原理
大数据·mapreduce
8K超高清10 小时前
中国8K摄像机：科技赋能文化传承新图景
大数据·人工智能·科技·物联网·智能硬件
2401_8712905812 小时前
MapReduce 的工作原理
大数据·mapreduce
SelectDB技术团队13 小时前
Apache Doris 2025 Roadmap：构建 GenAI 时代实时高效统一的数据底座
大数据·数据库·数据仓库·人工智能·ai·数据分析·湖仓一体
你觉得20513 小时前
浙江大学朱霖潮研究员：《人工智能重塑科学与工程研究》以蛋白质结构预测为例｜附PPT下载方法
大数据·人工智能·机器学习·ai·云计算·aigc·powerpoint
益莱储中国13 小时前
世界通信大会、嵌入式展及慕尼黑上海光博会亮点回顾
大数据
Loving_enjoy14 小时前
基于Hadoop的明星社交媒体影响力数据挖掘平台：设计与实现
大数据·hadoop·数据挖掘
浮尘笔记14 小时前
go-zero使用elasticsearch踩坑记：时间存储和展示问题
大数据·elasticsearch·golang·go
热门推荐
01我决定放弃搞 Java 了02DeepSeek各版本说明与优缺点分析03如何在WPS和Word/Excel中直接使用DeepSeek功能04【AUTOSAR 基础软件】Can模块详解（Can栈之驱动模块）05RAG 实践- Ollama+RagFlow 部署本地知识库06从零安装 LLaMA-Factory 微调 Qwen 大模型成功及所有的坑07Mac cursor设置jdk、Maven版本08本地化部署AI知识库：基于Ollama+DeepSeek+AnythingLLM保姆级教程09如何本地部署AI智能体平台，带你手搓一个AI Agent10纯离线部署本地知识库LLM大模型