This morning a friend of mine was grousing about his new X.509 certificates from a commercial CA. He'd put together a certificate bundle with the intermediate CA certs and it was working fine with both NGiNX and Apache, but somehow it wasn't working with his mail server (Exim) as reported by CheckTLS.com.

One never notices such things unless moving to new certificates, and when you move to new certificates is that time that you might decide to add wildcards, move around your SAN list, and other such modifications, so the answer to the usual debug question, "what changed", is going to be "a whole lot more than you really wanted".

I offered assistance and started poking at his SMTP service with comand-line OpenSSL. Many folks aren't aware that not only can OpenSSL act like a client (so long as you aren't using IPv6 - bug open for over 10 years still not fixed as of 1.0.2j - can't handle a hostname with only a quad-a record), but it can issue STARTTLS commands in the ways expected by SMTP, LDAP, XMPP, and other servers. For instance, to print out the details of the cert, one could do:

{% raw %} openssl s_client -starttls smtp -crlf -connect mail.example.com:25 </dev/null | openssl x509 -text -noout

Well, it turned out that I was getting some darned weird diags out - it seemed that the far end wasn't presenting a certificate at all.

{% raw %} rs@sandbox39.seastrom.com:~$ openssl s_client -starttls smtp -crlf -connect mail2.example.com:25 CONNECTED(00000004) 18446741324916576824:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:794: --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 244 bytes and written 340 bytes --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE

etc

{% endraw %}

After going down a couple of dead-ends (it wasn't the wildcard cert, or the signature type), it turned out that the certificate bundle he was using had not one, but two, intermediate certificates in it... but one wasn't a child of the other.

One had the Subject: {% raw %} Subject: C=BE, O=GlobalSign nv-sa, CN=GlobalSign Organization Validation CA - SHA256 - G2 {% endraw %} while the other had the Subject: {% raw %} Subject: OU=GlobalSign Root CA - R2, O=GlobalSign, CN=GlobalSign {% endraw %} These obviously aren't the same thing. They even had different issuers, which suggests that they were paths to different root certificates (my Mac has six certs with GlobalSign in their name in SystemRoots).

His (new, wildcard) certificate had: {% raw %} Issuer: C=BE, O=GlobalSign nv-sa, CN=GlobalSign Organization Validation CA - SHA256 - G2 {% endraw %} so I had him get rid of the other intermediate cert and create a bundle with only one intermediate cert, one that corresponded with his wildcard cert.

Success!

The theory (which is going to remain a theory because I'm too lazy to go digging in source code) is that both Apache and NGiNX are much more careful about data sanitization than Exim when they read in certificate files, which stands to reason because https:// is very common these days (and more likely to be set up by novices) whilst SMTP TLS is unfortunately comparatively rare, and Exim is a comparatively unusual unless you happen to be an expert or running out-of-the-box Debian.