How to Fix the Public Keys in Reply and Keystore Don't Match Error in Java

1. Understanding the Public Keys in Reply and Keystore Don't Match Error in Java
2. Verify CSR and Keystore Consistency
3. Fix the Public Keys in Reply and Keystore Don't Match Error in Java
4. Regenerate the CSR
5. Convert Certificate Formats
6. Examine Key Algorithm and Size
7. Debugging with Keytool
8. Conclusion

The Java Keytool utility is an essential tool for managing cryptographic keys, certificates, and keystores. However, encountering the error "keytool error: java.lang.Exception: Public keys in reply and keystore don't match" can be perplexing.

This article delves into the causes of this error and provides a detailed guide on various methods to rectify it. It also highlights the possible reasons that result in this error.

Further, we will learn about the different solutions we can use to fix this error.

Understanding the Public Keys in Reply and Keystore Don't Match Error in Java

The error message "keytool error: java.lang.Exception: Public keys in reply and keystore don't match" typically indicates a mismatch between the public key in the Certificate Signing Request (CSR) and the public key in the keystore. This discrepancy can arise from various issues, including key mismatches, certificate format inconsistencies, or errors during the certificate signing process.

We will try to access the web service hosted at port 443. The service provider shared three certificates: ABCD.cer, CCA_Certificate.cer, and CA_Certificate.cer.

We are required to add all of them to the keystore by creating a form chain for the SSL communication. The commands we followed from this article are given below in sequential order:

keytool -keystore npci_keystore.jks -genkey -alias npci_client

When you run this command, the keytool utility will prompt you to enter various pieces of information, such as the keystore password, distinguished name fields (like your name, organization, etc.), and the password for the key pair. The generated key pair and the associated self-signed certificate will be stored in the specified keystore file (npci_keystore.jks) with the given alias (npci_client).

keytool -import -keystore npci_keystore.jks -file CA_Certificate.cer -alias CARoot

When you run the above command, you may be prompted to enter the keystore password, depending on whether it’s password-protected. The certificate from the specified file (CA_Certificate.cer) will be imported into the keystore (npci_keystore.jks) with the given alias (CARoot).

keytool -import -keystore npci_keystore.jks -file CCA_Certificate.cer -alias CCARoot

The certificate from the specified file (CCA_Certificate.cer) will then be imported into the keystore (npci_keystore.jks) with the provided alias (CCARoot).

keytool -import -keystore npci_keystore.jks -file ABCD.cer -alias npci_client

The certificate from the specified file (ABCD.cer) will then be imported into the keystore (npci_keystore.jks) with the provided alias (npci_client). But after entering the keystore password, we got the error saying keytool error: java.lang.Exception: Public keys in reply and keystore don't match.

We get this problem for different reasons that are listed below:

1. We get this error when we try to generate a certificate with a different key pair.
2. This error occurs if we have used the same alias while importing a certificate and creating the JKS store.
3. Sometimes, installing certificates in the wrong order also results in this error.
4. We also go through this issue if the Root CA’s certificate is missing from the chain.

Now, the point is how to eradicate it. Let’s see that in the following section.

Verify CSR and Keystore Consistency

Before diving into more advanced solutions, it’s crucial to ensure that the public key in the CSR and the public key in the keystore are consistent. Use the following steps:

1. Check CSR Information:

   Verify the details in the CSR by examining the CSR file (.csr) or regenerating it if necessary. Ensure that the public key information in the CSR matches the intended key.

2. Verify Keystore Public Key:

   Inspect the public key stored in the keystore using the keytool command.

   keytool -list -keystore your_keystore_filename.jks
   

   Replace your_keystore_filename.jks with the actual filename of your keystore. Compare the keystore’s public key with the one in the CSR to ensure they match.

Fix the Public Keys in Reply and Keystore Don't Match Error in Java

In our scenario, the link we were using guides how we can create an SSL KeyStore for the server, which is not what we want to achieve. What we did was start with creating a new key pair.

Next, we added a trusted certificate to KeyStore, added another trusted certificate to KeyStore, and then tried to import the server’s SSL certificate for our key pair.

At this point, we are failing because the generated SSL certificate is for an entirely different key pair. The three certificates that we have include the following:

1. The web service’s SSL certificate
2. A CA certificate that signed SSL certificate
3. A root certificate that signed a CA

Now, we have to add the trust anchor to our TrustStore. By default, it is ${JAVA_HOME}/jre/lib/security/cacerts, with the outcome that our client accepts the SSL certificate of a web service.

Remember, during the SSL handshake, the SSL server sends an entire chain, excluding the root certificate, to a client. So, we must add a root certificate to our TrustStore as follows:

keytool -import -keystore ${JAVA_HOME}/jre/lib/security/cacerts -file CCA_Certificate.cer -alias CCARoot

Some additional steps are essential if a web service needs SSL client authentication. If we have never mentioned client authentication, then this is unnecessary.

This is how the error is fixed in our case, but there are other solutions we can try if we are in a different situation.

1. We need to generate the certificate again using the same actual key pair to eliminate this error.
2. There are some situations where the error is caused by using the same alias while creating the JKS store and importing the certificate. Then, we need to change the alias to resolve the error.
3. Make sure that the Root CA’s certificate is not missing from the chain.

Regenerate the CSR

If inconsistencies persist, consider regenerating the CSR. Follow these steps:

1. Revoke the Current Certificate (if applicable):

   If the certificate associated with the CSR has been issued, consider revoking it before generating a new CSR.

2. Generate a New CSR:

   Use the keytool command or another tool to generate a new CSR.

   keytool -certreq -keystore your_keystore_filename.jks -alias your_key_alias -file your_csr_filename.csr
   

   Replace your_key_alias with the actual alias of the key for which you want to generate the CSR, and replace your_csr_filename.csr with the desired filename for the CSR.

   To verify that the public key in the new CSR matches the one in the keystore, you can inspect the CSR using the keytool command or another tool. Here, we’ll use keytool to view the details:

   keytool -printcertreq -file your_csr_filename.csr
   

   This command will display information about the CSR, including the public key details. Look for the "Subject Public Key Algorithm" field and ensure that it matches the public key algorithm and key size from the keystore.

3. Submit the New CSR for Signing:

   Once you have verified the CSR’s public key against the one in the keystore, you can submit the CSR to a Certificate Authority (CA) for signing. The CA will provide you with a signed certificate that you can then import into your keystore.

Convert Certificate Formats

The issue may stem from differences in certificate formats. Convert the certificate to the appropriate format to align with the keystore.

Use openssl to convert the certificate format. For example, to convert from PEM to DER format, you might use:

openssl x509 -inform PEM -outform DER -in input.pem -out output.der

Replace input.pem with the path to your PEM certificate and output.der with the desired output path for the DER format.

Alternatively, if you are converting from DER to PEM, you can use:

openssl x509 -inform DER -outform PEM -in input.der -out output.pem

Replace input.der with the path to your DER certificate and output.pem with the desired output path for the PEM format.

Once the certificate is in the desired format, use the keytool command to import it into the keystore.

The basic syntax for importing a certificate into a keystore is as follows:

keytool -import -file converted_certificate_file -keystore your_keystore_filename.jks -alias your_alias

Replace converted_certificate_file with the path to your converted certificate file, your_keystore_filename.jks with the keystore filename, and your_alias with the desired alias for the certificate in the keystore. You may be prompted to enter the keystore password if it is password-protected.

For example, if you converted a PEM certificate to DER and want to import it into a keystore named "example.jks" with an alias "example_alias", the command might look like this:

keytool -import -file output.der -keystore example.jks -alias example_alias

Run the keytool -list command to verify that the certificate has been successfully imported into the keystore.

keytool -list -keystore your_keystore_filename.jks

Look for the entry associated with the specified alias (your_alias) in the keystore.

Examine Key Algorithm and Size

Incompatibility in key algorithms or sizes may lead to the error. Ensure consistency between the key algorithm and size in the CSR and the keystore.

If you have the CSR file, open it with a text editor to inspect its contents. Look for lines in the CSR that specify the key algorithm and size.

The information may look similar to:

-----BEGIN CERTIFICATE REQUEST-----
MIICiDCCAXACAQAwgYgxCzAJBgNVBAYTAkNOMRQwEgYDVQQIDAtNYXN0ZXJpY2Ex
...
-----END CERTIFICATE REQUEST-----

Within these lines, you should find information about the key algorithm and size used to generate the CSR.

Ensure that the key algorithm and size in the CSR match your expectations. If they do not, you may need to regenerate the CSR with the desired specifications.

Use the following keytool command to inspect the key algorithm and size in the keystore:

keytool -list -keystore your_keystore_filename.jks -v

Replace your_keystore_filename.jks with the actual filename of your keystore. You may be prompted to enter the keystore password if it is password-protected.

Look for the entry associated with the key alias in the keystore. The output will include information about the key pair, including the key algorithm and size.

For example:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: your_key_alias
Creation date: [timestamp]
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
...
Public Key Algorithm: RSA
Key size: 2048
...

Ensure that the key algorithm and size in the keystore match your expectations. If they do not, and you need to update them, you may need to generate a new key pair and update the keystore.

Debugging with Keytool

Utilize the keytool command for detailed diagnostics.

1. Check Keytool Debug Information:
   • Run the keytool command with the -v (verbose) and -list options to display detailed information.
   • Examine the output for any inconsistencies or errors.
2. Review Keytool Documentation:
   • Refer to the keytool documentation for additional options and troubleshooting tips.
   • Explore available options for resolving key-related issues.

Conclusion

The "java.lang.Exception: Public keys in reply and keystore don't match" error can be resolved through careful examination of the CSR, keystore, and associated certificates. This article has provided a comprehensive guide covering various methods, including verifying consistency, regenerating CSRs, converting certificate formats, ensuring key algorithm and size compatibility, and leveraging the keytool command for debugging.

As you navigate through these methods, it’s essential to document each step, pay close attention to error messages, and collaborate with the Certificate Authority if necessary. A systematic approach and thorough understanding of the keytool utility will empower you to diagnose and resolve this error effectively in your Java keystore management processes.