Letsencrypt certificate renewal behind http proxy fails with unexpected error: bad handshake

Published on March 18th 2019

Got the following alert from our OSSEC monitoring:

OSSEC HIDS Notification.
2019 Mar 18 12:47:52

Received From: (webserver) 192.168.100.54->/var/log/syslog
Rule: 1002 fired (level 2) -> "Unknown problem somewhere in the system."
Portion of the log(s):

Mar 18 12:47:51 webserver certbot[5644]: Attempting to renew cert from /etc/letsencrypt/renewal/www.example.com.conf produced an unexpected error: ("bad handshake: Error([('SSL routines', 'ssl3_get_record', 'wrong version number')],)",). Skipping.

I logged on to this server and was able to reproduce it when simply running certbot renew:

root@webserver ~ # certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/www.example.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Attempting to renew cert from /etc/letsencrypt/renewal/www.example.com.conf produced an unexpected error: ("bad handshake: Error([('SSL routines', 'ssl3_get_record', 'wrong version number')],)",). Skipping.

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/www.example.com/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)

This particular web server runs in a secured zone where outgoing connections all must go through a http proxy. And even then, only whitelisted domains are allowed to be accessed. Now I tried the same again with the https_proxy environment variable:

root@webserver ~ # export https_proxy=http://192.168.100.100:8888

root@webserver ~ # certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/www.example.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.example.com
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0001_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0001_csr-certbot.pem

-------------------------------------------------------------------------------
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/www.example.com/fullchain.pem
-------------------------------------------------------------------------------

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/www.example.com/fullchain.pem (success)

This time it worked! No surprise; without the http/https proxy, the outgoing connection would not work.

But one question remained: Why did certbot try to renew this certificate on its own? I did not set up a cron job on this web server.

This can be solved by examining the "certbot" package. It contains its own cronjob file and installs it into /etc/cron.d/:

root@webserver ~ # dpkg -S certbot|grep cron
certbot: /etc/cron.d/certbot

So I used this file and added the https_proxy variable to it:

0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(3600))' && export https_proxy=http://192.168.100.100:8888 && certbot -q renew

Next time the certificate has to be renewed, this should work out of the box.