文章目录
spark序列化报错
scala
org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 61186304. To avoid this, increase spark.kryoserializer.buffer.max value.
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:300)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:313)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
问题解决
the problem is that kryo does not have enough room in its buffer.
- You should be adjusting
spark.kryoserializer.buffer.max
in your properties file, - or use
--conf "spark.kryoserializer.buffer.max=128m"
in your spark-submit command.