Infiniroot Blog: We sometimes write, too.

Of course we cannot always share details about our work with customers, but nevertheless it is nice to show our technical achievements and share some of our implemented solutions.

NRPE: Error - Could not complete SSL handshake after check_nrpe was upgraded from 2.x to 3.x

Published on March 25th 2021


After a monitoring server, on which the nagios-plugin check_nrpe runs, was upgraded from Ubuntu 16.04 (xenial) to 18.04 (bionic), the check_nrpe plugin was upgraded, too.

NRPE SSL handshake errors after upgrading plugin to 3.2.1

Incompatible versions: nrpe 2.x <-> nrpe 3.x

On a Ubuntu 16.04, the nagios-nrpe-plugin package is installed in version 2.15. On Ubuntu 18.04, the plugin is upgraded to 3.2.1:

root@monitoring:~# dpkg -l|grep nrpe-plugin
ii  nagios-nrpe-plugin       3.2.1-1ubuntu1     amd64        Nagios Remote Plugin Executor Plugin

From that moment on, a couple of alerts appeared in our monitoring system (running Icinga 2). Target hosts still running on Ubuntu Xenial (and therefore running nagios-nrpe-server 2.15) would now return the following error:

root@monitoring:~# /usr/lib/nagios/plugins/check_nrpe -H xenial -c check_command
CHECK_NRPE: (ssl_err != 5) Error - Could not complete SSL handshake with xenial: 1

Obviously there is an (SSL) incompatibility between these two versions. Something similar also happened in the "parent" distribution, Debian (see Debian 9 Strech and Nagios NRPE command args and SSL compatibility) but this was fixed in the Debian 9 packages.

Update May 2nd 2021:

The same problem now happens in Debian 10 Buster, when NRPE was upgraded from 3.0.1-3+deb9u1 to 3.2.1-2.

Compile nrpe (server) from source

Luckily nrpe is a very leightweight application, and it's easy to compile the binary from source.

For the sake of completeness, here are the steps to manually compile nrpe from source.

ck@xenial:~$ sudo apt-get update
ck@xenial:~$ sudo apt-get install -y build-essential autoconf automake gcc libc6 libmcrypt-dev make libssl-dev wget openssl
ck@xenial:~$ wget --no-check-certificate -O nrpe.tar.gz https://github.com/NagiosEnterprises/nrpe/archive/nrpe-4.0.3.tar.gz
ck@xenial:~$ tar xzf nrpe.tar.gz
ck@xenial:~$ cd nrpe-nrpe-4.0.3/
ck@xenial:~$ ./configure --prefix=/usr --sysconfdir=/etc --libdir=/usr/lib/nagios --libexecdir=/usr/lib/nagios/plugins --localstatedir=/var --enable-ssl --with-ssl-lib=/usr/lib/$(DEB_HOST_MULTIARCH) --with-piddir=/var/run/nagios --enable-command-args
ck@xenial:~$ make nrpe

If all of these commands ran through without any error, this should result in a binary file called nrpe in the src directory:

ck@xenial:~/src/nrpe-nrpe-4.0.3$ ll src/nrpe
-rwxr-xr-x 1 root root 254552 Mar 25 08:50 src/nrpe*

Replace the nrpe (server) binary on the affected servers

This binary can now be placed on the Ubuntu 16.04 servers still running nagios-nrpe-server 2.15. Simply stop nagios-nrpe-server, create a backup of /usr/sbin/nrpe, replace /usr/sbin/nrpe with the new binary and finally start nagios-nrpe-server again:

root@xenial:~# wget https://www.claudiokuenzler.com/downloads/nrpe/nrpe-4.0.3-xenial
root@xenial:~# chmod 755 nrpe-4.0.3-xenial
root@xenial:~# cp /usr/sbin/nrpe{,.backup}
root@xenial:~# systemctl stop nagios-nrpe-server
root@xenial:~# cp nrpe-4.0.3-xenial /usr/sbin/nrpe
cp: overwrite '/usr/sbin/nrpe'? y
root@xenial:~# systemctl start nagios-nrpe-server

As you can see, the nrpe binary for Ubuntu Xenial is prepared and available publicly. But for security reasons, you should compile your own binary.

Note: A way better solution would be to create a deb package for the new nrpe version for Xenial but the effort would have been much higher and all my Xenial systems will be either upgraded or shut down this year.

Does it work now?

Right after replacing the binary and restarting nagios-nrpe-server the checks recovered. A manual check confirms that the compatibility between the plugin (3.2.1) and the server (4.0.3) is working:

root@monitoring:~# /usr/lib/nagios/plugins/check_nrpe -H xenial
NRPE v4.0.3

root@monitoring:~# /usr/lib/nagios/plugins/check_nrpe -H xenial -c check_load -a "10,9,8" "20,15,12"
OK - load average: 0.07, 0.02, 0.00|load1=0.070;10.000;20.000;0; load5=0.020;9.000;15.000;0; load15=0.000;8.000;12.000;0;

Update: Workaround using newer check_nrpe with NRPE 2 daemon

Updated May 2nd 2021

The same error (CHECK_NRPE: (ssl_err != 5) Error - Could not complete SSL handshake) is now also seen after upgrading Icinga 2's OS from Debian Stretch to Buster. If the target system is still using NRPE 2, the following workaround can be applied: Switch to non-ssl connection.

On the target, where NRPE runs as daemon, adjust the nrpe startup script (e.g. /etc/init.d/nagios-nrpe-server) and add the command parameter -n to it:

[...]
case "$1" in
  start)
    if [ "$INETD" = 1 ]; then
        exit 1
    fi
    log_daemon_msg "Starting $DESC" "$NAME"
    start_daemon $NICENESS $DAEMON -c $CONFIG -d -n $DAEMON_OPTS
    log_end_msg $?
    ;;
[...]

Then restart nrpe on the target.

On the monitoring server, where check_nrpe plugin is launched, use the -n parameter in the command line, too.

If you are using Icinga 2 with apply rules, the following adjustment (Handle old NRPE versions) can help you:

root@icinga2:~# cat /etc/icinga2/zones.d/global-templates/applyrules/cpuload.conf
apply Service "CPU Load" {

  display_name = "CPU Load"
  import "generic-service"
  check_command = "nrpe"
  vars.nrpe_command = "check_load"

  # Handle old NRPE versions (SSL incompatibility)
  if (host.vars.distro.version == "6") {
    vars.nrpe_no_ssl = true
        vars.nrpe_version_2 = true
  }


  assign where host.address && host.vars.os == "Linux"
  ignore where host.vars.applyignore.cpuload == true
}

Of course this requires, that the actual host(s) still running an old NRPE version have the custom variable "vars.distro.version" set.

Disabling the encryption is really only considered a workaround. You should make sure that: