Python调用pyspark报错整理

赫加青空2024-02-05 11:08

前言

Pycharm配置了SSH服务器和Anaconda的python解释器，如果没有配置可参考 大数据单机学习环境搭建(8)Linux单节点Anaconda安装和Pycharm连接

Pycharm执行的脚本

执行如下 pyspark_model.py 的python脚本，构建SparkSession来执行sparksql

python 复制代码 
"""
    脚本名称：Pycharm使用pyspark测试
    功能：Pycharm远程执行sparksql
"""
from pyspark.sql import SparkSession
import os

os.environ['SPARK_HOME'] = '/opt/spark'
os.environ['JAVA_HOME'] = '/opt/jdk1.8'

spark = SparkSession.builder \
    .appName('pyspark_conda') \
    .master("yarn") \
    .config("spark.sql.warehouse.dir", "hdfs://bigdata01:8020/user/hive/warehouse") \
    .config("hive.metastore.uris", "thrift://bigdata01:9083") \
    .enableHiveSupport() \
    .getOrCreate()

spark.sql('select * from hostnames limit 10;').show()

spark.stop()

报错一：pyspark版本不匹配

例如我当前集群环境Spark3.0.0，python的pyspark3.5.0，没有指定版本默认下载了最新的

报错信息 [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number., 具体如下：

复制代码 
ssh://slash@bigdata01:22/opt/python3/bin/python3 -u /home/slash/etl/dwtool/pyspark/pyspark_script/pyspark_model.py
JAVA_HOME is not set
Traceback (most recent call last):
  File "/home/slash/etl/dwtool/pyspark/pyspark_script/pyspark_model.py", line 7, in <module>
    spark = SparkSession.builder \
  File "/opt/python3/lib/python3.8/site-packages/pyspark/sql/session.py", line 497, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 515, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 201, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 436, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
  File "/opt/python3/lib/python3.8/site-packages/pyspark/java_gateway.py", line 107, in launch_gateway
    raise PySparkRuntimeError(
pyspark.errors.exceptions.base.PySparkRuntimeError: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.

如果坚持不更换python的pyspark版本，即使像报错2已经指定了JAVA_HOME 依然会有其他报错。例如下方报错 Py4JError ，所以最彻底的方法是替换pyspark版本与spark版本一致

复制代码 
Traceback (most recent call last):
  File "/home/slash/etl/dwtool/pyspark/pyspark_script/pyspark_model.py", line 7, in <module>
    spark = SparkSession.builder \
  File "/opt/python3/lib/python3.8/site-packages/pyspark/sql/session.py", line 497, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 515, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 203, in __init__
    self._do_init(
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 316, in _do_init
    self._jvm.PythonUtils.getPythonAuthSocketTimeout(self._jsc)
  File "/opt/python3/lib/python3.8/site-packages/py4j/java_gateway.py", line 1549, in __getattr__
    raise Py4JError(
py4j.protocol.Py4JError: org.apache.spark.api.python.PythonUtils.getPythonAuthSocketTimeout does not exist in the JVM

报错二：JAVA_HOME指定不成功

python的pyspark已经重装3.0.0版本（下载时指定版本 pip install pyspark==3.0.0），报错信息 Java gateway process exited before sending its port number., JAVA_HOME is not set 具体如下：

复制代码 
ssh://slash@bigdata01:22/opt/python3/bin/python3 -u /home/slash/etl/dwtool/pyspark/pyspark_script/pyspark_model.py
JAVA_HOME is not set
Traceback (most recent call last):
  File "/home/slash/etl/dwtool/pyspark/pyspark_script/pyspark_model.py", line 7, in <module>
    spark = SparkSession.builder \
  File "/opt/python3/lib/python3.8/site-packages/pyspark/sql/session.py", line 186, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 371, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 128, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 320, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
  File "/opt/python3/lib/python3.8/site-packages/pyspark/java_gateway.py", line 105, in launch_gateway
    raise Exception("Java gateway process exited before sending its port number")
Exception: Java gateway process exited before sending its port number

指定内容如下：

python 复制代码 
# pyspark3.5.0指定了 SPARK_HOME JAVA_HOME还是会报错
# pyspark3.0.0指定后成功运行
os.environ['SPARK_HOME'] = '/opt/spark'
os.environ['JAVA_HOME'] = '/opt/jdk1.8'

报错三：python版本问题

最开始安装的最新版的anaconda环境，其中python3.11，安装pyspark3.0.0也会报错 TypeError: code() argument 13 must be str, not int，具体内容如下：

复制代码 
ssh://slash@bigdata01:22/opt/anaconda3/bin/python3.11 -u /home/slash/etl/dwtool/pyspark/pyspark_script/pyspark_model.py
Traceback (most recent call last):
  File "/home/slash/etl/dwtool/pyspark/pyspark_script/pyspark_model.py", line 1, in <module>
    from pyspark.sql import SparkSession
  File "/opt/anaconda3/lib/python3.11/site-packages/pyspark/__init__.py", line 51, in <module>
    from pyspark.context import SparkContext
  File "/opt/anaconda3/lib/python3.11/site-packages/pyspark/context.py", line 30, in <module>
    from pyspark import accumulators
  File "/opt/anaconda3/lib/python3.11/site-packages/pyspark/accumulators.py", line 97, in <module>
    from pyspark.serializers import read_int, PickleSerializer
  File "/opt/anaconda3/lib/python3.11/site-packages/pyspark/serializers.py", line 71, in <module>
    from pyspark import cloudpickle
  File "/opt/anaconda3/lib/python3.11/site-packages/pyspark/cloudpickle.py", line 209, in <module>
    _cell_set_template_code = _make_cell_set_template_code()
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/lib/python3.11/site-packages/pyspark/cloudpickle.py", line 172, in _make_cell_set_template_code
    return types.CodeType(
           ^^^^^^^^^^^^^^^
TypeError: code() argument 13 must be str, not int

删除 /opt/anaconda3的文件夹后，重新安装了 Anaconda3-2021.05-Linux-x86_64.sh 版本的anaconda，其中python3.8，利用pyspark3.0.0第三方库操作spark3.0.0的计算引擎构建SparkSession，执行sparksql成功。

声明：本文所载信息不保证准确性和完整性。文中所述内容和意见仅供参考，不构成实际商业建议，如有雷同纯属巧合。

相关推荐
老王IT11 小时前
C#语言基础语法介绍
开发语言·c#
校园安全大脑平台11 小时前
校园安全平台过等保二级，开发阶段要注意什么
开发语言·安全·php
只睡四小时11 小时前
端侧推理：face-api.js 首屏 92% 是白下载的
开发语言·javascript·人脸识别·ecmascript·tensorflow.js·face-api.js·端侧推理
yuzhiboyouye11 小时前
com.fasterxml.jackson.annotation 里最常用高频的注解，可以按“控制字段、控制格式、控制结构、控制流程”来记
java·开发语言
HAPPY酷11 小时前
C++ 内存泄漏排查：从裸 new 到 RAII 修复，附 ASan 实战
开发语言·c++
伤心的梧桐叶12 小时前
Picset与创客贴比等待时间，没做完的任务不能从表里消失
python
十年Java程序媛12 小时前
Java 面试常问 this、super：构造方法调用顺序避坑
java·开发语言·spring boot·面试·个人开发
没有气的汽水12 小时前
一次 SQS 消费者静默退出的排查：问题最后竟然出在日志里
python
用户4500301402312 小时前
物业催缴系统为什么总在"冷却"这一步翻车：账期建模与规则命中的几个取舍
python
Web极客码12 小时前
决策模型 + LLM：用 Python 拆解浏览器自动化工作流
java·python·自动化
热门推荐
01国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）02如何新建文件夹？ 电脑新建文件夹的4种方法03GitHub 镜像站点04历年考研数学一、数学二、数学三真题试卷及答案PDF05几个好用的ip纯净度检测网站06ai安全工具:CyberStrikeAI安装部署与使用07苹果iPA游戏软件资源下载网站08昇腾910B系列芯片完全指南：四款型号对比、服务器选型与大模型部署实战09乙二醇水溶液物性参数10「全语录+省流版」邵艾伦 × 孙宇晨：聊风口、聊个人IP、聊年轻人如何抓住 AI 时代的机会