why doesn't java send the client certificate during SSL handshake?

I'm trying to connect to a secure webservice.

I was getting a handshake failure even though my keystore and truststore have been set correctly.

After several days of frustration, endless googling and asking everyone around I found out that the only problem was that java chose not to send the client certificate to the server during the handshake.

Specifically:

1. Server requested a client certificate (CN=RootCA) - i.e. "give me a cert that is signed by the root CA"
2. Java looked into the keystore and only found my client certificate which is signed by the "SubCA", which in turn is issued by the "RootCA". It didn't bother to look into the truststore...duh OK I guess
3. Sadly when I tried to add the "SubCA" certificate to the keystore, that didn't help at all. I did check if the certificates get loaded into the keystore. They do, but the KeyManager ignores all certificates except the client one.
4. All of the above leads to the fact that java decides it doesn't have any certificates that satisfy the server's request and sends nothing...tadaaa handshake failure :-(

My questions:

1. Is it possible that I added the "SubCA" certificate to the keystore in a manner that "broke the certificate chain" or something so that the KeyManager only loads the client certificate and ignores the rest? (Chrome and openssl manage to figure that out so why can't java? - note that the "SubCA" cert is always presented separately as the trusted authority so Chrome apparently correctly packs it along with the client cert during handshake)
2. Is this a formal "configuration issue" on the server side? The server is a third party. I would expect the server to request a certificate signed by the "SubCA" authority since that's what they provided us with. I suspect that the fact that this works in Chrome and openssl is because they are "less restrictive" and java just goes "by the book" and fails.

I did manage to put together a dirty workaround for this, but I'm not very happy about it so I'll be glad if anyone can clarify this one for me.

It's possible that you may have imported the intermediate CA certificate into the keystore without associating it with the entry where you have your client certificate and its private key. You should be able to see this using keytool -v -list -keystore store.jks. If you only get one certificate per alias entry, they're not together.

You would need to import your certificate and its chain together into the keystore alias that has your private key.

To find out which keystore alias has the private key, use keytool -list -keystore store.jks (I'm assuming JKS store type here). This will tell you something like this:

Your keystore contains 1 entry

myalias, Feb 15, 2012, PrivateKeyEntry, 
Certificate fingerprint (MD5): xxxxxxxx

Here, the alias is myalias. If you use -v in addition to this, you should see Alias Name: myalias.

If you don't have it separately already, export your client certificate from the keystore:

keytool -exportcert -rfc -file clientcert.pem -keystore store.jks -alias myalias

This should give you a PEM file.

Using a text editor (or cat), prepare file (let's call it bundle.pem) with that client certificate and the intermediate CA certificate (and possibly the root CA certificate itself if you want), so that the client-certificate is at the beginning and its issuer cert is just under.

This should look like:

-----BEGIN CERTIFICATE-----
MIICajCCAdOgAwIBAgIBAjANBgkqhkiG9w0BAQUFADA7MQswCQYDVQQGEwJVSzEa
....
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIICkjCCAfugAwIBAgIJAKm5bDEMxZd7MA0GCSqGSIb3DQEBBQUAMDsxCzAJBgNV
....
-----END CERTIFICATE-----

Now, import this bundle back together into the alias where your private key is:

keytool -importcert -keystore store.jks -alias myalias -file bundle.pem

As an add here, you can use %> openssl s_client -connect host.example.com:443 and see the dump and check that all the main cert is valid against the client. You are looking for this at the bottom of the output. Verify return code: 0 (ok)

If you add -showcerts it will dump all the info of the keychain that was sent along with the host certificate, which is what you loaded into your keychain.

Most of the solutions I've seen revolve around using the keytool and none of them matched my case.

Here is a very brief description: I've got a PKCS12 (.p12) which works fine in Postman with disabled certificate verification, however programmatically I always ended up getting server error "400 Bad Request" / "No required SSL certificate was sent".

The reason was a missing TLS extension SNI (Server Name Indication) and following is the solution.

---

Adding an extension/parameter to SSL Context

After SSLContext init, add the following:

SSLSocketFactory factory = sslContext.getSocketFactory();
	try (SSLSocket socket = (SSLSocket) factory.createSocket(host, port)) {
		SSLParameters sslParameters = socket.getSSLParameters();
		sslParameters.setServerNames(Collections.singletonList(new SNIHostName(hostName)));
		socket.setSSLParameters(sslParameters);
		socket.startHandshake();
	}

---

Full HTTP Client class for this case (NOT FOR PRODUCTION)

Note 1: SSLContextException and KeyStoreFactoryException simply extend RuntimeException.

Note 2: Certificate validations are disabled, this example was intended for dev use only.

Note 3: Disabling hostname verification was not required in my case, but I included it as a commented line

import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.ssl.SSLContexts;

import javax.net.ssl.*;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.net.URL;
import java.security.*;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.Collections;
import java.util.Objects;

public class SecureClientBuilder {

	private String host;
	private int port;
    private boolean keyStoreProvided;
	private String keyStorePath;
	private String keyStorePassword;

	public SecureClientBuilder withSocket(String host, int port) {
		this.host = host;
		this.port = port;
		return this;
	}

	public SecureClientBuilder withKeystore(String keyStorePath, String keyStorePassword) {
        this.keyStoreProvided = true;
		this.keyStorePath = keyStorePath;
		this.keyStorePassword = keyStorePassword;
		return this;
	}

	public CloseableHttpClient build() {
		SSLContext sslContext = keyStoreProvided
				? getContextWithCertificate()
				: SSLContexts.createDefault();

		SSLConnectionSocketFactory sslSocketFactory =
				new SSLConnectionSocketFactory(sslContext);

		return HttpClients.custom()
				.setSSLSocketFactory(sslSocketFactory)
                //.setSSLHostnameVerifier(NoopHostnameVerifier.INSTANCE)
				.build();
	}

	private SSLContext getContextWithCertificate() {
		try {
			// Generate TLS context with specified KeyStore and
			SSLContext sslContext = SSLContext.getInstance("TLS");
			sslContext.init(getKeyManagerFactory().getKeyManagers(), new TrustManager[]{getTrustManager()}, new SecureRandom());

			SSLSocketFactory factory = sslContext.getSocketFactory();
			try (SSLSocket socket = (SSLSocket) factory.createSocket(host, port)) {
				SSLParameters sslParameters = socket.getSSLParameters();
				sslParameters.setServerNames(Collections.singletonList(new SNIHostName(host)));
				socket.setSSLParameters(sslParameters);
				socket.startHandshake();
			}

			return sslContext;
		} catch (NoSuchAlgorithmException | KeyManagementException | IOException e) {
			throw new SSLContextException("Could not create an SSL context with specified keystore.\nError: " + e.getMessage());
		}
	}

	private KeyManagerFactory getKeyManagerFactory() {
		try (FileInputStream fileInputStream = getResourceFile(keyStorePath)) {
			// Read specified keystore
			KeyStore keyStore = KeyStore.getInstance("PKCS12");
			keyStore.load(fileInputStream, keyStorePassword.toCharArray());

			// Init keystore manager
			KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance("SunX509");
			keyManagerFactory.init(keyStore, keyStorePassword.toCharArray());
			return keyManagerFactory;
		} catch (NoSuchAlgorithmException | CertificateException | UnrecoverableKeyException | IOException | KeyStoreException e) {
			throw new KeyStoreFactoryException("Could not read the specified keystore.\nError: " + e.getMessage());
		}
	}

    // Bypasses error: "unable to find valid certification path to requested target"
	private TrustManager getTrustManager() {
		return new X509TrustManager() {
			@Override
			public void checkClientTrusted(java.security.cert.X509Certificate[] arg0, String arg1) {
			}

			@Override
			public void checkServerTrusted(java.security.cert.X509Certificate[] arg0, String arg1) {
			}

			@Override
			public java.security.cert.X509Certificate[] getAcceptedIssuers() {
				return new X509Certificate[0];
			}
		};
	}

	private FileInputStream getResourceFile(String keyStorePath) throws FileNotFoundException {
		URL resourcePath = getClass().getClassLoader().getResource(keyStorePath);
		return new FileInputStream(resourcePath.getFile());
	}

}

---

Using the client builder above

Note 1: keystore (.p12) is looked for in "resources" folder.

Note 2: Header "Host" is set to avoid server error "400 - Bad Request".

String hostname = "myHost";
int port = 443;
String keyStoreFile = "keystore.p12";
String keyStorePass = "somepassword";
String endpoint = String.format("https://%s:%d/endpoint", host, port);

CloseableHttpClient apacheClient = new SecureClientBuilder()
		.withSocket(hostname, port)
		.withKeystore(keyStoreFile, keyStorePass)
		.build();

HttpGet get = new HttpGet(endpoint);
get.setHeader("Host", hostname + ":" + port);
CloseableHttpResponse httpResponse = apacheClient.execute(get);

assert httpResponse.getStatusLine().getStatusCode() == 200;

---

Reference docs

https://docs.oracle.com/en/java/javase/11/security/java-secure-socket-extension-jsse-reference-guide.html

The thing is when using client cert signed by an intermediate one you need to include the intermediate one in your trustore so java can find it. Either solo or as a bundle with the root issuing ca.