pycharm-pyspark 环境安装

sunxunyong2025-01-08 15:18

1、环境准备：java、scala、pyspark、python-anaconda、pycharm

vi ~/.bash_profile

export SCALA_HOME=/Users/xunyongsun/Documents/scala-2.13.0

export PATH= P A T H : PATH: PATH:SCALA_HOME/bin

export SPARK_HOME=/Users/xunyongsun/Documents/spark-3.5.4-bin-hadoop3

export PATH= P A T H : PATH: PATH:SPARK_HOME/bin

export PYTHONPATH= S P A R K H O M E / p y t h o n : SPARK_HOME/python: SPARKHOME/python:SPARK_HOME/python/lib/py4j-0.10.9.7-src.zip

export PATH= S P A R K H O M E / p y t h o n : SPARK_HOME/python: SPARKHOME/python:PATH

source ~/.bash_profile

scala -version

java -version

pyspark

pip -V

python -V

2、创建project
3、选中项目,点击Pycharm->Settings...->点击 Project:xxx:->Project Structure

4、配置SPARK_HOME

写好pyspark脚本后，在运行python脚本之前，要配置下SPARK_HOME

5、conda 的env环境增加pyspark引用

6、运行脚本