We sometimes write.

Of course we cannot always share details about our work with customers, but nevertheless it is nice to show our achievements and share some solutions.

Letsencrypt certificate renewal behind http proxy fails with unexpected error: bad handshake

Published on March 18th 2019 - see original post


Got the following alert from our OSSEC monitoring:

OSSEC HIDS Notification.
2019 Mar 18 12:47:52

Received From: (webserver) 192.168.100.54->/var/log/syslog
Rule: 1002 fired (level 2) -> "Unknown problem somewhere in the system."
Portion of the log(s):

Mar 18 12:47:51 webserver certbot[5644]: Attempting to renew cert from /etc/letsencrypt/renewal/www.example.com.conf produced an unexpected error: ("bad handshake: Error([('SSL routines', 'ssl3_get_record', 'wrong version number')],)",). Skipping.

I logged on to this server and was able to reproduce it when simply running certbot renew:

root@webserver ~ # certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/www.example.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Attempting to renew cert from /etc/letsencrypt/renewal/www.example.com.conf produced an unexpected error: ("bad handshake: Error([('SSL routines', 'ssl3_get_record', 'wrong version number')],)",). Skipping.

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/www.example.com/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)

This particular web server runs in a secured zone where outgoing connections all must go through a http proxy. And even then, only whitelisted domains are allowed to be accessed. Now I tried the same again with the https_proxy environment variable:

root@webserver ~ # export https_proxy=http://192.168.100.100:8888

root@webserver ~ # certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/www.example.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.example.com
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0001_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0001_csr-certbot.pem

-------------------------------------------------------------------------------
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/www.example.com/fullchain.pem
-------------------------------------------------------------------------------

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/www.example.com/fullchain.pem (success)

This time it worked! No surprise; without the http/https proxy, the outgoing connection would not work.

But one question remained: Why did certbot try to renew this certificate on its own? I did not set up a cron job on this web server.

This can be solved by examining the "certbot" package. It contains its own cronjob file and installs it into /etc/cron.d/:

root@webserver ~ # dpkg -S certbot|grep cron
certbot: /etc/cron.d/certbot

So I used this file and added the https_proxy variable to it:

0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(3600))' && export https_proxy=http://192.168.100.100:8888 && certbot -q renew

Next time the certificate has to be renewed, this should work out of the box.