Python调用pyspark报错整理

赫加青空2024-02-05 11:08

前言

Pycharm配置了SSH服务器和Anaconda的python解释器，如果没有配置可参考 大数据单机学习环境搭建(8)Linux单节点Anaconda安装和Pycharm连接

Pycharm执行的脚本

执行如下 pyspark_model.py 的python脚本，构建SparkSession来执行sparksql

python 复制代码 
"""
    脚本名称：Pycharm使用pyspark测试
    功能：Pycharm远程执行sparksql
"""
from pyspark.sql import SparkSession
import os

os.environ['SPARK_HOME'] = '/opt/spark'
os.environ['JAVA_HOME'] = '/opt/jdk1.8'

spark = SparkSession.builder \
    .appName('pyspark_conda') \
    .master("yarn") \
    .config("spark.sql.warehouse.dir", "hdfs://bigdata01:8020/user/hive/warehouse") \
    .config("hive.metastore.uris", "thrift://bigdata01:9083") \
    .enableHiveSupport() \
    .getOrCreate()

spark.sql('select * from hostnames limit 10;').show()

spark.stop()

报错一：pyspark版本不匹配

例如我当前集群环境Spark3.0.0，python的pyspark3.5.0，没有指定版本默认下载了最新的

报错信息 [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number., 具体如下：

复制代码 
ssh://slash@bigdata01:22/opt/python3/bin/python3 -u /home/slash/etl/dwtool/pyspark/pyspark_script/pyspark_model.py
JAVA_HOME is not set
Traceback (most recent call last):
  File "/home/slash/etl/dwtool/pyspark/pyspark_script/pyspark_model.py", line 7, in <module>
    spark = SparkSession.builder \
  File "/opt/python3/lib/python3.8/site-packages/pyspark/sql/session.py", line 497, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 515, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 201, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 436, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
  File "/opt/python3/lib/python3.8/site-packages/pyspark/java_gateway.py", line 107, in launch_gateway
    raise PySparkRuntimeError(
pyspark.errors.exceptions.base.PySparkRuntimeError: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.

如果坚持不更换python的pyspark版本，即使像报错2已经指定了JAVA_HOME 依然会有其他报错。例如下方报错 Py4JError ，所以最彻底的方法是替换pyspark版本与spark版本一致

复制代码 
Traceback (most recent call last):
  File "/home/slash/etl/dwtool/pyspark/pyspark_script/pyspark_model.py", line 7, in <module>
    spark = SparkSession.builder \
  File "/opt/python3/lib/python3.8/site-packages/pyspark/sql/session.py", line 497, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 515, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 203, in __init__
    self._do_init(
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 316, in _do_init
    self._jvm.PythonUtils.getPythonAuthSocketTimeout(self._jsc)
  File "/opt/python3/lib/python3.8/site-packages/py4j/java_gateway.py", line 1549, in __getattr__
    raise Py4JError(
py4j.protocol.Py4JError: org.apache.spark.api.python.PythonUtils.getPythonAuthSocketTimeout does not exist in the JVM

报错二：JAVA_HOME指定不成功

python的pyspark已经重装3.0.0版本（下载时指定版本 pip install pyspark==3.0.0），报错信息 Java gateway process exited before sending its port number., JAVA_HOME is not set 具体如下：

复制代码 
ssh://slash@bigdata01:22/opt/python3/bin/python3 -u /home/slash/etl/dwtool/pyspark/pyspark_script/pyspark_model.py
JAVA_HOME is not set
Traceback (most recent call last):
  File "/home/slash/etl/dwtool/pyspark/pyspark_script/pyspark_model.py", line 7, in <module>
    spark = SparkSession.builder \
  File "/opt/python3/lib/python3.8/site-packages/pyspark/sql/session.py", line 186, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 371, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 128, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 320, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
  File "/opt/python3/lib/python3.8/site-packages/pyspark/java_gateway.py", line 105, in launch_gateway
    raise Exception("Java gateway process exited before sending its port number")
Exception: Java gateway process exited before sending its port number

指定内容如下：

python 复制代码 
# pyspark3.5.0指定了 SPARK_HOME JAVA_HOME还是会报错
# pyspark3.0.0指定后成功运行
os.environ['SPARK_HOME'] = '/opt/spark'
os.environ['JAVA_HOME'] = '/opt/jdk1.8'

报错三：python版本问题

最开始安装的最新版的anaconda环境，其中python3.11，安装pyspark3.0.0也会报错 TypeError: code() argument 13 must be str, not int，具体内容如下：

复制代码 
ssh://slash@bigdata01:22/opt/anaconda3/bin/python3.11 -u /home/slash/etl/dwtool/pyspark/pyspark_script/pyspark_model.py
Traceback (most recent call last):
  File "/home/slash/etl/dwtool/pyspark/pyspark_script/pyspark_model.py", line 1, in <module>
    from pyspark.sql import SparkSession
  File "/opt/anaconda3/lib/python3.11/site-packages/pyspark/__init__.py", line 51, in <module>
    from pyspark.context import SparkContext
  File "/opt/anaconda3/lib/python3.11/site-packages/pyspark/context.py", line 30, in <module>
    from pyspark import accumulators
  File "/opt/anaconda3/lib/python3.11/site-packages/pyspark/accumulators.py", line 97, in <module>
    from pyspark.serializers import read_int, PickleSerializer
  File "/opt/anaconda3/lib/python3.11/site-packages/pyspark/serializers.py", line 71, in <module>
    from pyspark import cloudpickle
  File "/opt/anaconda3/lib/python3.11/site-packages/pyspark/cloudpickle.py", line 209, in <module>
    _cell_set_template_code = _make_cell_set_template_code()
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/lib/python3.11/site-packages/pyspark/cloudpickle.py", line 172, in _make_cell_set_template_code
    return types.CodeType(
           ^^^^^^^^^^^^^^^
TypeError: code() argument 13 must be str, not int

删除 /opt/anaconda3的文件夹后，重新安装了 Anaconda3-2021.05-Linux-x86_64.sh 版本的anaconda，其中python3.8，利用pyspark3.0.0第三方库操作spark3.0.0的计算引擎构建SparkSession，执行sparksql成功。

声明：本文所载信息不保证准确性和完整性。文中所述内容和意见仅供参考，不构成实际商业建议，如有雷同纯属巧合。

相关推荐
风痕天际16 分钟前
Pytorch开发教程1——CUDA安装
人工智能·pytorch·python
梦幻通灵19 分钟前
Java中finally失效的几种情况【持续更新】
java·开发语言
月落归舟24 分钟前
SpringMVC 多种响应返回方式
java·开发语言
长不胖的路人甲37 分钟前
二叉排序树（BST）Java 完整实现 + 删除思路详解
java·开发语言·算法
geovindu38 分钟前
python: Breadth First Search Algorithm and Depth First Search Algorithm
开发语言·后端·python·算法·搜索算法
程序员果子42 分钟前
CrewAI ：当 Agent 学会团队协作
人工智能·git·python·多智能体·agent框架
玉鸯43 分钟前
向量检索不是记忆：Agent 记忆的三层进化
python·agent
大模型码小白44 分钟前
Java 部署：Jenkins Pipeline 构建 Java 项目（自动化）
java·人工智能·python·机器学习
runafterhit1 小时前
python基础语法命令（C程序员刷leetcode）
c语言·python·leetcode
paeamecium1 小时前
【PAT甲级真题】- Kuchiguse (20)
数据结构·c++·python·算法·pat考试·pat
热门推荐
01GitHub 镜像站点02如何新建文件夹？ 电脑新建文件夹的4种方法032026年7月AI圈大地震：GPT-5.6被政府限制、Claude入驻Slack、Anthropic自研芯片04AI科技热点日报 | 2026年07月01日05国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）06AI 编程 IDE 全景解析 2026：Agent 全面接管开发链路072026 国产 AI 大模型横评：DeepSeek、通义千问、Kimi、文心一言、星火、豆包谁更能打？08幻兽帕鲁 - 服务器管理员权限与 GM 命令完全指南092026 年 AI 大模型 & AI 编程工具实战全总结10微信历史版本含下载地址（ Windows PC | 安卓 | MAC ）及设置微信不更新