38、spark读取hudi报错:java.io.NotSerializableException: org.apache.hadoop.fs.Path

场景:spark.table()的方式读取hudi映射的hive表。

开源组件版本:

spark 2.4.5_2.11

hudi 0.10.0

hive 3.1.0

hadoop 2.8.5

报错代码:

scala 复制代码
spark.table("dwd.dwd_card_menu_a_1d")
.show(false)

报错日志:

java 复制代码
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Failed to serialize task 0, not attempting to retry it. Exception during serialization: java.io.NotSerializableException: org.apache.hadoop.fs.Path
Serialization stack:
	- object not serializable (class: org.apache.hadoop.fs.Path, value: obs://xxxxx/user/hive/warehouse/dwd.db/dwd_card_menu_a_1d)
	- element of array (index: 0)
	- array (class [Ljava.lang.Object;, size 1)
	- field (class: scala.collection.mutable.WrappedArray$ofRef, name: array, type: class [Ljava.lang.Object;)
	- object (class scala.collection.mutable.WrappedArray$ofRef, WrappedArray(obs://xxxxxx/user/hive/warehouse/dwd.db/dwd_card_menu_a_1d))
	- writeObject data (class: org.apache.spark.rdd.ParallelCollectionPartition)
	- object (class org.apache.spark.rdd.ParallelCollectionPartition, org.apache.spark.rdd.ParallelCollectionPartition@691)
	- field (class: org.apache.spark.scheduler.ResultTask, name: partition, type: interface org.apache.spark.Partition)
	- object (class org.apache.spark.scheduler.ResultTask, ResultTask(0, 0))
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1891)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1879)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1878)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1878)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:927)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:927)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:927)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2112)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2061)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2050)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:738)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:990)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:385)
	at org.apache.spark.rdd.RDD.collect(RDD.scala:989)
	at org.apache.spark.api.java.JavaRDDLike$class.collect(JavaRDDLike.scala:361)
	at org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45)
	at org.apache.hudi.client.common.HoodieSparkEngineContext.map(HoodieSparkEngineContext.java:100)
	at org.apache.hudi.metadata.FileSystemBackedTableMetadata.getAllPartitionPaths(FileSystemBackedTableMetadata.java:81)
	at org.apache.hudi.common.fs.FSUtils.getAllPartitionPaths(FSUtils.java:291)

org.apache.hadoop.fs.Path 未实现java.io.serializabe。 因为 Hudi-0.10.0 在 Spark 2.4.5 下会把 Path 对象放进 ParallelCollectionRDD 的闭包,而 Path 不可序列化。

解决办法:

很简单,在sparkConf中设置spark的序列化为:KryoSerializer

scala 复制代码
new SparkConf()
    .set("spark.serializer", classOf[KryoSerializer].getName)
相关推荐
lang201509288 分钟前
打造专属Spring Boot Starter
java·spring boot·后端
曹牧26 分钟前
C#:数组不能使用Const修饰符
java·数据结构·算法
YA33341 分钟前
java设计模式六、装饰器模式
java·设计模式·装饰器模式
回忆是昨天里的海1 小时前
k8s集群-节点间通信之安装kube-flannel插件
java·docker·kubernetes
信仰_2739932432 小时前
Mybatis-Spring重要组件介绍
java·spring·mybatis
盖世英雄酱581362 小时前
java深度调试【第二章通过堆栈分析性能瓶颈】
java·后端
没有bug.的程序员2 小时前
AOP 原理深剖:动态代理与 CGLIB 字节码增强
java·spring·aop·动态代理·cglib
2401_837088502 小时前
ResponseEntity - Spring框架的“标准回复模板“
java·前端·spring
lang201509282 小时前
Spring Boot RSocket:高性能异步通信实战
java·spring boot·后端
默默coding的程序猿3 小时前
1.北京三维天地公司-实施实习生
java·sql·技术支持·面经·实施·实施工程师·三维天地