Apache Spark: How to use pyspark with Python 3

PythonPython 3.xApache Spark

Python Problem Overview

I built Spark 1.4 from the GH development master, and the build went through fine. But when I do a bin/pyspark I get the Python 2.7.9 version. How can I change this?

Python Solutions

Solution 1 - Python

Just set the environment variable:

export PYSPARK_PYTHON=python3

in case you want this to be a permanent change add this line to pyspark script.

Solution 2 - Python

PYSPARK_PYTHON=python3 
./bin/pyspark

If you want to run in in IPython Notebook, write:

PYSPARK_PYTHON=python3 
PYSPARK_DRIVER_PYTHON=ipython 
PYSPARK_DRIVER_PYTHON_OPTS="notebook" 
./bin/pyspark

If python3 is not accessible, you need to pass path to it instead.

Bear in mind that the current documentation (as of 1.4.1) has outdate instructions. Fortunately, it has been patched.

Solution 3 - Python

1,edit profile :vim ~/.profile

2,add the code into the file: export PYSPARK_PYTHON=python3

3, execute command : source ~/.profile

4, ./bin/pyspark

Solution 4 - Python

Have a look into the file. The shebang line is probably pointed to the 'env' binary which searches the path for the first compatible executable.

You can change python to python3. Change the env to directly use hardcoded the python3 binary. Or execute the binary directly with python3 and omit the shebang line.

Solution 5 - Python

For Jupyter Notebook, edit spark-env.sh file as shown below from command line

$ vi $SPARK_HOME/conf/spark-env.sh

Goto the bottom of the file and copy paste these lines

export PYSPARK_PYTHON=python3
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"

Then, simply run following command to start pyspark in notebook

$ pyspark
Categories
Recommended Python Solutions
Recommended Python 3.x Solutions
Recommended Apache Spark Solutions

Previous Solution:

How can I remove apps from Organizer in Xcode?

Xcode

Next Solution:

How to sort a json file by keys and values of those keys in jq

JsonSortingJq

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestiontchakravartyView Question on Stackoverflow
Solution 1 - PythonRtik88View Answer on Stackoverflow
Solution 2 - PythonPiotr MigdalView Answer on Stackoverflow
Solution 3 - PythonyanghView Answer on Stackoverflow
Solution 4 - PythonrfkortekaasView Answer on Stackoverflow
Solution 5 - Pythonoya163View Answer on Stackoverflow