Spark异常:Kryo serialization failed: Buffer overflow.
1、问题描述
SparkSQL任务报错如下:
java
org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: xxx. To avoid this, increase spark.kryoserializer.buffer.max value.
根据报错信息可知,Spark在Kryo序列化时发生缓存溢出
2、问题解决
根据提示信息,可以适当增加spark.kryoserializer.buffer.max
的值,例如256m或512m,该参数的默认值为128m
更多信息参考:https://study.sf.163.com/documents/read/service_support/dsc-p-a-0067