Java Gateway Process Exited Before Sending Its Port Number
This tutorial demonstrates the Java gateway process exited before sending the driver its port number
error in Java.
Java Gateway Process Exited Before Sending the Driver Its Port Number
PySpark
is used to write Spark
applications using Python APIs and provides a shell for interactively analyzing the data in a distributed environment.
While running the PySpark
application in PySpark Shell
, spark-submit
, or even from spyder
the error Pyspark: Exception: Java gateway process exited before sending the driver its port number
can occur.
If you want to run PySpark
on your Windows, Mac, or Linux, you must install Java on your machine because PySpark
is usually considered to work as a Python framework. The reason for this error to occur is that either you haven’t set the JAVA_HOME
or PYSPARK_SUBMIT_ARGS
.
Here are the solutions to solve the Pyspark: Exception: Java gateway process exited before sending the driver its port number
error.
Install Open JDK or Oracle JDK
You must wonder why you need Java to run PySpark
. It is because the Spark
is basically written in Scala
, but later the industry adopted its API, so the PySpark
was released for Python using the Py4J
, as Py4J
is a Java Library that is why Java is required to run PySpark
.
Py4J
is integrated into PySpark
, allowing Python to interface with JVM objects.
We can use the following command to install the OpenJDK.
# Installing Open JDK
sudo add-apt-repository ppa:openjdk-r/ppa
sudo apt-get update
sudo apt-get install openjdk-11-jdk
# Installing Oracle JDK
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer
Once the JDK is installed, you need to set the JAVA_HOME
path and follow the steps below.
-
Find your JDK path; if Java is installed, the path will be
/usr/lib/jvm/java-1.x.x-openjdk
. -
Now run the following command to set the
JAVA_HOME
path.export JAVA_HOME=/usr/lib/jvm/java-1.x.x-openjdk
-
The command will set the
JAVA_HOME
path to verify runecho $JAVA_HOME
, showing the Java installation path.
The above commands are for the Linux environments for Windows. The installers can be downloaded from here:
To set the JAVA_HOME
at Windows, follow the steps below.
-
Search
Environment Variables
in the Windows search bar. -
Open
Edit the system environment variables
. -
Click the
Environment Variables
button to open environment variables. -
In the system variables, click
New
. -
The name will be
JAVA_HOME
. -
The value will be the installation path, for example,
C:\Program Files\Java\jdk-17.0.2
. -
After putting the name and value, click
OK
andApply changes
.
Set PYSPARK_SUBMIT_ARGS
Most of the time, the error Pyspark: Exception: Java gateway process exited before sending the driver its port number
occurs when the PYSPARK_SUBMIT_ARGS
is not set. Use the following command to set the PYSPARK_SUBMIT_ARGS
.
export PYSPARK_SUBMIT_ARGS="--master local[3] pyspark-shell"
The above command will set the PYSPARK_SUBMIT_ARGS
using the master
. Following both solutions will solve the problem.
Sheeraz is a Doctorate fellow in Computer Science at Northwestern Polytechnical University, Xian, China. He has 7 years of Software Development experience in AI, Web, Database, and Desktop technologies. He writes tutorials in Java, PHP, Python, GoLang, R, etc., to help beginners learn the field of Computer Science.
LinkedIn Facebook