Java Gateway Process Exited Before Sending Its Port Number

Sheeraz Gul Aug 12, 2022
Java Gateway Process Exited Before Sending Its Port Number

This tutorial demonstrates the Java gateway process exited before sending the driver its port number error in Java.

Java Gateway Process Exited Before Sending the Driver Its Port Number

PySpark is used to write Spark applications using Python APIs and provides a shell for interactively analyzing the data in a distributed environment.

While running the PySpark application in PySpark Shell, spark-submit, or even from spyder the error Pyspark: Exception: Java gateway process exited before sending the driver its port number can occur.

If you want to run PySpark on your Windows, Mac, or Linux, you must install Java on your machine because PySpark is usually considered to work as a Python framework. The reason for this error to occur is that either you haven’t set the JAVA_HOME or PYSPARK_SUBMIT_ARGS.

Here are the solutions to solve the Pyspark: Exception: Java gateway process exited before sending the driver its port number error.

Install Open JDK or Oracle JDK

You must wonder why you need Java to run PySpark. It is because the Spark is basically written in Scala, but later the industry adopted its API, so the PySpark was released for Python using the Py4J, as Py4J is a Java Library that is why Java is required to run PySpark.

Py4J is integrated into PySpark, allowing Python to interface with JVM objects.

We can use the following command to install the OpenJDK.

# Installing Open JDK
sudo add-apt-repository ppa:openjdk-r/ppa
sudo apt-get update
sudo apt-get install openjdk-11-jdk

# Installing Oracle JDK
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

Once the JDK is installed, you need to set the JAVA_HOME path and follow the steps below.

  • Find your JDK path; if Java is installed, the path will be /usr/lib/jvm/java-1.x.x-openjdk.
  • Now run the following command to set the JAVA_HOME path.
    export JAVA_HOME=/usr/lib/jvm/java-1.x.x-openjdk
    
  • The command will set the JAVA_HOME path to verify run echo $JAVA_HOME, showing the Java installation path.

The above commands are for the Linux environments for Windows. The installers can be downloaded from here:

  1. OpenJDK
  2. Oracle

To set the JAVA_HOME at Windows, follow the steps below.

  • Open Edit the system environment variables.
  • Click the Environment Variables button to open environment variables.
  • In the system variables, click New.
  • The name will be JAVA_HOME.
  • The value will be the installation path, for example, C:\Program Files\Java\jdk-17.0.2.
  • After putting the name and value, click OK and Apply changes.

Set PYSPARK_SUBMIT_ARGS

Most of the time, the error Pyspark: Exception: Java gateway process exited before sending the driver its port number occurs when the PYSPARK_SUBMIT_ARGS is not set. Use the following command to set the PYSPARK_SUBMIT_ARGS.

export PYSPARK_SUBMIT_ARGS="--master local[3] pyspark-shell"

The above command will set the PYSPARK_SUBMIT_ARGS using the master. Following both solutions will solve the problem.

Author: Sheeraz Gul
Sheeraz Gul avatar Sheeraz Gul avatar

Sheeraz is a Doctorate fellow in Computer Science at Northwestern Polytechnical University, Xian, China. He has 7 years of Software Development experience in AI, Web, Database, and Desktop technologies. He writes tutorials in Java, PHP, Python, GoLang, R, etc., to help beginners learn the field of Computer Science.

LinkedIn Facebook

Related Article - Java Exception