38、spark读取hudi报错:java.io.NotSerializableException: org.apache.hadoop.fs.Path

场景:spark.table()的方式读取hudi映射的hive表。

开源组件版本:

spark 2.4.5_2.11

hudi 0.10.0

hive 3.1.0

hadoop 2.8.5

报错代码:

scala 复制代码
spark.table("dwd.dwd_card_menu_a_1d")
.show(false)

报错日志:

java 复制代码
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Failed to serialize task 0, not attempting to retry it. Exception during serialization: java.io.NotSerializableException: org.apache.hadoop.fs.Path
Serialization stack:
	- object not serializable (class: org.apache.hadoop.fs.Path, value: obs://xxxxx/user/hive/warehouse/dwd.db/dwd_card_menu_a_1d)
	- element of array (index: 0)
	- array (class [Ljava.lang.Object;, size 1)
	- field (class: scala.collection.mutable.WrappedArray$ofRef, name: array, type: class [Ljava.lang.Object;)
	- object (class scala.collection.mutable.WrappedArray$ofRef, WrappedArray(obs://xxxxxx/user/hive/warehouse/dwd.db/dwd_card_menu_a_1d))
	- writeObject data (class: org.apache.spark.rdd.ParallelCollectionPartition)
	- object (class org.apache.spark.rdd.ParallelCollectionPartition, org.apache.spark.rdd.ParallelCollectionPartition@691)
	- field (class: org.apache.spark.scheduler.ResultTask, name: partition, type: interface org.apache.spark.Partition)
	- object (class org.apache.spark.scheduler.ResultTask, ResultTask(0, 0))
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1891)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1879)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1878)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1878)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:927)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:927)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:927)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2112)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2061)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2050)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:738)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:990)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:385)
	at org.apache.spark.rdd.RDD.collect(RDD.scala:989)
	at org.apache.spark.api.java.JavaRDDLike$class.collect(JavaRDDLike.scala:361)
	at org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45)
	at org.apache.hudi.client.common.HoodieSparkEngineContext.map(HoodieSparkEngineContext.java:100)
	at org.apache.hudi.metadata.FileSystemBackedTableMetadata.getAllPartitionPaths(FileSystemBackedTableMetadata.java:81)
	at org.apache.hudi.common.fs.FSUtils.getAllPartitionPaths(FSUtils.java:291)

org.apache.hadoop.fs.Path 未实现java.io.serializabe。 因为 Hudi-0.10.0 在 Spark 2.4.5 下会把 Path 对象放进 ParallelCollectionRDD 的闭包,而 Path 不可序列化。

解决办法:

很简单,在sparkConf中设置spark的序列化为:KryoSerializer

scala 复制代码
new SparkConf()
    .set("spark.serializer", classOf[KryoSerializer].getName)
相关推荐
hello 早上好2 小时前
Spring Boot 核心启动机制与配置原理剖析
java·spring boot·后端
2023框框2 小时前
方法器 --- 策略模式(Strategy Pattern)
java·策略模式
Brookty3 小时前
【Java学习】定时器Timer(源码详解)
java·开发语言·学习·多线程·javaee
编啊编程啊程4 小时前
gRPC从0到1系列【6】
java·rpc·kafka·dubbo·nio
IT毕设梦工厂4 小时前
大数据毕业设计选题推荐-基于大数据的全球经济指标数据分析与可视化系统-Hadoop-Spark-数据可视化-BigData
大数据·hadoop·数据分析·spark·毕业设计·源码·bigdata
宸津-代码粉碎机4 小时前
Redis 进阶:跳出缓存局限!7 大核心场景的原理与工程化实践
java·人工智能·redis·python
极客先躯4 小时前
Spring Statemachine 架构详解
java·spring·架构
ccccczy_5 小时前
Java微服务容器化与 Kubernetes 编排实战:从 Docker 多阶段构建到云原生弹性扩展
java·docker·kubernetes·springboot·microservices·cloudnative·containerization
沉木渡香6 小时前
VSCode中Java开发环境配置的三个层级(Windows版)1-3
java·windows·vscode