How to Fix the Javax.Net.SSL.SSLHandshakeException Error

1. SSL Handshakes in Java
2. Fix the SSLHandshakeException Because of Missing Server Certificate
3. Fix the SSLHandShakeException Because of Untrusted Server Certificate
4. Fix the SSLHandShakeException Because of Incorrect Certificate
5. Fix the SSLHandShakeException Because of Incompatible SSL Version and Cipher Suite

This tutorial demonstrates the javax.net.ssl.SSLHandshakeException error in Java.

SSL Handshakes in Java

The SSL Handshakes are used for the client and server to establish trust and logistics required to secure the connection over the internet. There are typical steps in SSL Handshake operations which are needed to be followed:

1. First, the client will provide the list of all possible cipher suites and SSL versions.
2. The server will then agree on the particular cipher suite and SSL version, which will respond with a certificate.
3. Then, the client will extract the public key from the given certificate and respond with the new encrypted pre-master key.
4. Then, the server will use the private key to decrypt the pre-master key.
5. Then, the client and server will compute the shared secret using the pre-master key together.
6. Finally, the client and server will exchange messages which confirm the successful encryption and decryption of the shared secret.

The SSL Handshakes have two types. The first is the one-way SSL, which leaves the server to trust all the clients, and the second is the two-way SSL in which the client and server must accept each other’s certificates.

After understanding the SSL Handshakes, we can now discuss the SSLHandShakeException in detail. There are two scenarios of SSLHandShakeException, which are given below.

Fix the SSLHandshakeException Because of Missing Server Certificate

If an SSL Handshake operation when the client is connecting to the server did not receive any certificate, it would throw the SSLHandShakeException as mentioned below:

Exception in thread "main" javax.net.ssl.SSLHandshakeException:
  Received fatal alert: handshake_failure

To solve this issue, make sure you follow all the steps above. This problem occurs when the Keystore or the system properties are not entered properly.

Keystores are the certificates provided by the authorities, or we can also create our Keystores by using the keytool functionality of JDK. Here is an example for the Keystore:

$ keytool -genkey -keypass password \
                  -storepass password \
                  -keystore Server_Keystore.jks

The above keytool code is written in the Keystore file. Now the keytool can be used to extract a public certificate from the Keystore file which was generated above:

$ keytool -export -storepass password \
                  -file NewServer.cer \
                  -keystore Server_Keystore.jks

The above code will export the public certificate from the Keystore as a file NewServer.cer. Now, we can add it to the Truststore for the client:

$ keytool -import -v -trustcacerts \
                     -file NewServer.cer \
                     -keypass password \
                     -storepass password \
                     -keystore Client_Truststore.jks

Now the Keystore for the server and Truststore for the client are generated. We can pass them as system properties to the server with a command:

-Djavax.net.ssl.keyStore=Client_Keystore.jks -Djavax.net.ssl.keyStorePassword=password

It is necessary for the system property. The Keystore file path must be absolute or place the Keystore file in the same directory from where the command is invoked.

The relative paths are not supported. Once you follow this process, the missing certificate error will be solved, and there will be no more SSLHandShakeException.

Fix the SSLHandShakeException Because of Untrusted Server Certificate

The other reason for SSLHandShakeException is an untrusted server certificate. When a server is using a self-signed certificate that is not signed by authorities, it will throw the following error:

Exception in thread "main" javax.net.ssl.SSLHandshakeException:
  sun.security.validator.ValidatorException:
  PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException:
  unable to find valid certification path to requested target

This exception will be thrown whenever the certificate is signed by any entity other than the default store. The default Truststore in JDK ships the information about common certificates in use.

This issue can be solved by forcing the client to trust the certificate presented by the server. We need to use the Truststore we generated above and pass them as system properties to the client:

-Djavax.net.ssl.trustStore=Client_Truststore.jks -Djavax.net.ssl.trustStorePassword=password

This will solve the exception, but it is not an ideal situation. In an ideal situation, we can use the self-signed certificate, which should be certified by the Certificate Authority (CA), then the client can trust them by default.

Fix the SSLHandShakeException Because of Incorrect Certificate

A handshake can also fail because of an incorrect certificate. When a certificate is not created properly, it will throw the SSLHandShakeException:

Exception in thread "main" javax.net.ssl.SSLHandshakeException:
    java.security.cert.CertificateException:
    No name matching localhost found

To check if the certificate is created properly, run the following command:

keytool -v -list -keystore Server_Keystore.jks

The above command will show the details of the Keystore owner:

...
Owner: CN=localhost, OU=technology, O=delftstack, L=city, ST=state, C=xx
..

The owner’s CN must match the server’s CN, and if it doesn’t match, it will throw the same exception shown above as it is generated because of the different CN.

Fix the SSLHandShakeException Because of Incompatible SSL Version and Cipher Suite

While an SSL handshake operation, it is possible that there will be various cryptographic protocols like different versions of SSL, TLS, etc. While the client and server must agree on cryptographic protocols and versions on a handshake, the SSL is superseded by the TLS for its cryptographic strength.

Now, for example, if the server is using the protocol SSL3, and the client is using the protocol TLS1.3, both cannot agree on the cryptographic protocol, and it will throw the SSLHandShakeException:

Exception in thread "main" javax.net.ssl.SSLHandshakeException:
  No appropriate protocol (protocol is disabled or cipher suites are inappropriate)

To solve this issue, we must verify that client and server are using either the same or compatible cryptographic protocols.

Similarly, it is also necessary to have the compatible Cipher Suite. While a handshake, the client provides the list of ciphers, and the server will select a cipher to use.

If the server cannot select a suitable cipher, the code will throw the following SSLHandShakeException:

Exception in thread "main" javax.net.ssl.SSLHandshakeException:
  Received fatal alert: handshake_failure

Normally the client and server use a variety of cipher suites; that is why this error can occur. The error occurs because a server has chosen a very selective cipher.

To avoid this issue, the server uses a list of selective ciphers, which is also good for security.