openssl s_client -cert: Proving a client certificate was sent to the server

Background

I am stuck in a finger-pointing match with a service provider with an API protected by SSL server and client certificates.

• I have generated a CSR, obtained a certificate from a public CA (GoDaddy in this case) and provided the certificate and CA chain to the service provider.
• They have supposedly loaded the the CA and my client certificate into their gateway.
• I am working with the most basic level tests using openssl s_client -connect ... -cert ... -key ...
• The provider tells me that their logs suggest my requests do not include a client SSL certificate at all.
• Strangely, the proper CA issuer for my certificate does appear in list of "Acceptable client certificate CA names" provided during the SSL handshake.
• For reference, a self-signed certificate I created and provided to them for testing does in fact work properly.

A sample (failed) request

[shell ~]$ openssl s_client -connect host:443 -cert cert_and_key.pem -key cert_and_key.pem -state -quiet
CONNECTED(00000003)
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
depth=2 **SNIP**
verify return:1
depth=1 **SNIP**
verify return:1
depth=0 **SNIP**
verify return:1
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server certificate request A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client certificate A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write certificate verify A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL3 alert read:fatal:unknown CA
SSL_connect:failed in SSLv3 read finished A
140011313276744:error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca:s3_pkt.c:1197:SSL alert number 48
140011313276744:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:184:

My reading of the SSL3 alert read:fatal:unknown CA error is that the server does not recognize the issuer of the certificate I am (in fact) providing. However, the provider "assures" me that the CA certificates are loaded properly and I have been unable to convince them otherwise.

Question

So, putting other (extensive) troubleshooting steps aside, what I'd really like to know is:

Is there some output available from openssl s_client that conclusively shows that a client certificate wasn't just requested by the server, but in fact was transmitted to the server during the SSL handshake?

I have experimented with the -state, -msg, -debug and -trace options, but don't have the background necessary to interpret the output.

EJP's answer suggests that the sample output I provided is proof enough with the write client certificate A, but this output appears regardless of whether the -cert options was used on the command line or not, so it's not indicative that a certificate was sent.

In order to verify a client certificate is being sent to the server, you need to analyze the output from the combination of the -state and -debug flags.

First as a baseline, try running

$ openssl s_client -connect host:443 -state -debug

You'll get a ton of output, but the lines we are interested in look like this:

SSL_connect:SSLv3 read server done A
write to 0x211efb0 [0x21ced50] (12 bytes => 12 (0xC))
0000 - 16 03 01 00 07 0b 00 00-03                        .........
000c - <SPACES/NULS>
SSL_connect:SSLv3 write client certificate A

What's happening here:

• The -state flag is responsible for displaying the end of the previous section:

    SSL_connect:SSLv3 read server done A  
  

This is only important for helping you find your place in the output.

• Then the -debug flag is showing the raw bytes being sent in the next step:

    write to 0x211efb0 [0x21ced50] (12 bytes => 12 (0xC))
    0000 - 16 03 01 00 07 0b 00 00-03                        .........
    000c - <SPACES/NULS>
  

• Finally, the -state flag is once again reporting the result of the step that -debug just echoed:

    SSL_connect:SSLv3 write client certificate A
  

So in other words: s_client finished reading data sent from the server, and sent 12 bytes to the server as (what I assume is) a "no client certificate" message.

---

If you repeat the test, but this time include the -cert and -key flags like this:

$ openssl s_client -connect host:443 \
   -cert cert_and_key.pem \
   -key cert_and_key.pem  \
   -state -debug

your output between the "read server done" line and the "write client certificate" line will be much longer, representing the binary form of your client certificate:

SSL_connect:SSLv3 read server done A
write to 0x7bd970 [0x86d890] (1576 bytes => 1576 (0x628))
0000 - 16 03 01 06 23 0b 00 06-1f 00 06 1c 00 06 19 31   ....#..........1
(*SNIP*)
0620 - 95 ca 5e f4 2f 6c 43 11-                          ..^%/lC.
SSL_connect:SSLv3 write client certificate A

The 1576 bytes is an excellent indication on its own that the cert was transmitted, but on top of that, the right-hand column will show parts of the certificate that are human-readable: You should be able to recognize the CN and issuer strings of your cert in there.

I know this is an old question but it does not yet appear to have an answer. I've duplicated this situation, but I'm writing the server app, so I've been able to establish what happens on the server side as well. The client sends the certificate when the server asks for it and if it has a reference to a real certificate in the s_client command line. My server application is set up to ask for a client certificate and to fail if one is not presented. Here is the command line I issue:

Yourhostname here -vvvvvvvvvv s_client -connect <hostname>:443 -cert client.pem -key cckey.pem -CAfile rootcert.pem -cipher ALL:!ADH:!LOW:!EXP:!MD5:@STRENGTH -tls1 -state

When I leave out the "-cert client.pem" part of the command the handshake fails on the server side and the s_client command fails with an error reported. I still get the report "No client certificate CA names sent" but I think that has been answered here above.

The short answer then is that the server determines whether a certificate will be sent by the client under normal operating conditions (s_client is not normal) and the failure is due to the server not recognizing the CA in the certificate presented. I'm not familiar with many situations in which two-way authentication is done although it is required for my project.

You are clearly sending a certificate. The server is clearly rejecting it.

The missing information here is the exact manner in which the certs were created and the way in which the provider loaded the cert, but that is probably all wrapped up by now.