Infiniroot Blog: We sometimes write, too.

Of course we cannot always share details about our work with customers, but nevertheless it is nice to show our technical achievements and share some solutions.

NRPE: Error - Could not complete SSL handshake after check_nrpe was upgraded from 2.x to 3.x

Published on March 25th 2021

After a monitoring server, on which the nagios-plugin check_nrpe runs, was upgraded from Ubuntu 16.04 (xenial) to 18.04 (bionic), the check_nrpe plugin was upgraded, too.

NRPE SSL handshake errors after upgrading plugin to 3.2.1

Incompatible versions: nrpe 2.x <-> nrpe 3.x

On a Ubuntu 16.04, the nagios-nrpe-plugin package is installed in version 2.15. On Ubuntu 18.04, the plugin is upgraded to 3.2.1:

root@monitoring:~# dpkg -l|grep nrpe-plugin
ii  nagios-nrpe-plugin       3.2.1-1ubuntu1     amd64        Nagios Remote Plugin Executor Plugin

From that moment on, a couple of alerts appeared in our monitoring system (running Icinga 2). Target hosts still running on Ubuntu Xenial (and therefore running nagios-nrpe-server 2.15) would now return the following error:

root@monitoring:~# /usr/lib/nagios/plugins/check_nrpe -H xenial -c check_command
CHECK_NRPE: (ssl_err != 5) Error - Could not complete SSL handshake with xenial: 1

Obviously there is an (SSL) incompatibility between these two versions. Something similar also happened in the "parent" distribution, Debian (see Debian 9 Strech and Nagios NRPE command args and SSL compatibility) but this was fixed in the Debian 9 packages.

Compile nrpe (server) from source

Luckily nrpe is a very leightweight application, and it's easy to compile the binary from source.

For the sake of completeness, here are the steps to manually compile nrpe from source.

ck@xenial:~$ sudo apt-get update
ck@xenial:~$ sudo apt-get install -y build-essential autoconf automake gcc libc6 libmcrypt-dev ck@xenial:~$ make libssl-dev wget openssl
ck@xenial:~$ wget --no-check-certificate -O nrpe.tar.gz
ck@xenial:~$ tar xzf nrpe.tar.gz
ck@xenial:~$ cd nrpe-nrpe-4.0.3/
ck@xenial:~$ ./configure --prefix=/usr --sysconfdir=/etc --libdir=/usr/lib/nagios --libexecdir=/usr/lib/nagios/plugins --localstatedir=/var --enable-ssl --with-ssl-lib=/usr/lib/$(DEB_HOST_MULTIARCH) --with-piddir=/var/run/nagios --enable-command-args
ck@xenial:~$ make nrpe

If all of these commands ran through without any error, this should result in a binary file called nrpe in the src directory:

ck@xenial:~/src/nrpe-nrpe-4.0.3$ ll src/nrpe
-rwxr-xr-x 1 root root 254552 Mar 25 08:50 src/nrpe*

Replace the nrpe (server) binary on the affected servers

This binary can now be placed on the Ubuntu 16.04 servers still running nagios-nrpe-server 2.15. Simply stop nagios-nrpe-server, create a backup of /usr/sbin/nrpe, replace /usr/sbin/nrpe with the new binary and finally start nagios-nrpe-server again:

root@xenial:~# wget
root@xenial:~# chmod 755 nrpe-4.0.3-xenial
root@xenial:~# cp /usr/sbin/nrpe{,.backup}
root@xenial:~# systemctl stop nagios-nrpe-server
root@xenial:~# cp nrpe-4.0.3-xenial /usr/sbin/nrpe
cp: overwrite '/usr/sbin/nrpe'? y
root@xenial:~# systemctl start nagios-nrpe-server

As you can see, the nrpe binary for Ubuntu Xenial is prepared and available publicly. But for security reasons, you should compile your own binary.

Note: A way better solution would be to create a deb package for the new nrpe version for Xenial but the effort would have been much higher and all my Xenial systems will be either upgraded or shut down this year.

Does it work now?

Right after replacing the binary and restarting nagios-nrpe-server the checks recovered. A manual check confirms that the compatibility between the plugin (3.2.1) and the server (4.0.3) is working:

root@monitoring:~# /usr/lib/nagios/plugins/check_nrpe -H xenial
NRPE v4.0.3

root@monitoring:~# /usr/lib/nagios/plugins/check_nrpe -H xenial -c check_load -a "10,9,8" "20,15,12"
OK - load average: 0.07, 0.02, 0.00|load1=0.070;10.000;20.000;0; load5=0.020;9.000;15.000;0; load15=0.000;8.000;12.000;0;