Of course we cannot always share details about our work with customers, but nevertheless it is nice to show our technical achievements and share some solutions.
In a recent article we already wrote about an Icinga 2 problem when upgrading Debian 9 (Stretch) to 10 (Buster) and Icinga packages would not be upgraded. A similar problem also happens on an upgraded Ubuntu 18.04 (Bionic) machine, when the previous system was 16.04 (Xenial).
In this case, a standard apt-get upgrade was launched to install the newest packages on an Ubuntu 18.04 (Bionic). This machine was recently upgraded from Ubuntu 16.04 (Xenial) but has been running on Bionic for a while now.
During apt was running the upgrade, a newer Icinga 2 version was also downloaded and installed:
Unpacking icinga2 (2.12.4-1.bionic) over (2.12.1-1.xenial) ...
Preparing to unpack .../50-icinga2-bin_2.12.4-1.bionic_amd64.deb ...
Unpacking icinga2-bin (2.12.4-1.bionic) over (2.12.1-1.xenial) ...
Preparing to unpack .../51-icinga2-common_2.12.4-1.bionic_all.deb ...
Unpacking icinga2-common (2.12.4-1.bionic) over (2.12.1-1.xenial) ...
Preparing to unpack .../52-icinga2-doc_2.12.4-1.bionic_all.deb ...
Unpacking icinga2-doc (2.12.4-1.bionic) over (2.12.1-1.xenial) ...
Note that the previous package (2.12.1) was still a xenial package. This means that after the OS upgrade, the Icinga packages for Xenial still remained installed. Again, that's the exact same problem as mentioned in the mentioned article (Icinga 2 with MySQL IDO fails to start after Debian release upgrade). The reason is that apt handles the versioning numeric and alphabetical. With the same 2.12.1 versions available on both xenial and bionic, the letter x is considered "newer" as it comes later in the alphabet.
At the end of the upgrade process, apt returned an error:
Errors were encountered while processing:
icinga2-common
icinga2-bin
icinga2
E: Sub-process /usr/bin/dpkg returned an error code (1)
The reason for this can be spotted, when checking the current status of the Icinga 2 daemon:
root@satellite:~# systemctl status icinga2
- icinga2.service - Icinga host/service/network monitoring system
Loaded: loaded (/lib/systemd/system/icinga2.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/icinga2.service.d
|-limits.conf
Active: failed (Result: exit-code) since Fri 2021-05-28 11:33:17 CEST; 1min 13s ago
Main PID: 11778 (code=exited, status=127)
May 28 11:33:17 satellite systemd[1]: Starting Icinga host/service/network monitoring system...
May 28 11:33:17 satellite icinga2[11778]: /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2: symbol lookup error: /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2: undefined symbol: _ZNK5boost16re_detail_10670031cpp_regex_traits_implementati
May 28 11:33:17 satellite systemd[1]: icinga2.service: Main process exited, code=exited, status=127/n/a
May 28 11:33:17 satellite systemd[1]: icinga2.service: Failed with result 'exit-code'.
May 28 11:33:17 satellite systemd[1]: Failed to start Icinga host/service/network monitoring system.
The symbol lookup error mentions boost in the error message. Let's take a closer look at the installed boost packages:
root@satellite:~# dpkg -l|grep libboost
ii libboost-context1.67.0-icinga:amd64 1.67.0-13.1.xenial amd64 provides a sort of cooperative multitasking on a single thread
ii libboost-coroutine1.67.0-icinga:amd64 1.67.0-13.1.xenial amd64 provides a sort of cooperative multitasking on a single thread
ii libboost-filesystem1.58.0:amd64 1.58.0+dfsg-5ubuntu3.1 amd64 filesystem operations (portable paths, iteration over directories, etc) in C++
ii libboost-filesystem1.65.1:amd64 1.65.1+dfsg-0ubuntu5 amd64 filesystem operations (portable paths, iteration over directories, etc) in C++
ii libboost-filesystem1.67.0-icinga:amd64 1.67.0-13.1.xenial amd64 filesystem operations (portable paths, iteration over directories, etc) in C++
ii libboost-iostreams1.58.0:amd64 1.58.0+dfsg-5ubuntu3.1 amd64 Boost.Iostreams Library
ii libboost-iostreams1.65.1:amd64 1.65.1+dfsg-0ubuntu5 amd64 Boost.Iostreams Library
ii libboost-program-options1.67.0-icinga:amd64 1.67.0-13.1.xenial amd64 program options library for C++
ii libboost-regex1.67.0-icinga:amd64 1.67.0-13.1.xenial amd64 regular expression library for C++
ii libboost-system1.58.0:amd64 1.58.0+dfsg-5ubuntu3.1 amd64 Operating system (e.g. diagnostics support) library
ii libboost-system1.65.1:amd64 1.65.1+dfsg-0ubuntu5 amd64 Operating system (e.g. diagnostics support) library
ii libboost-system1.67.0-icinga:amd64 1.67.0-13.1.xenial amd64 Operating system (e.g. diagnostics support) library
ii libboost-thread1.67.0-icinga:amd64 1.67.0-13.1.xenial amd64 portable C++ multi-threading
The libbost packages obviously coming from the Icinga installation (appended with -icinga) are still based on Xenial. No wonder the now upgraded Icinga 2 package would have troubles with a "wrong" libboost package.
As the libboost packages in the Icinga 2 repositories are still the same version (1.67.0-13.1) an upgrade of these packages won't do it. Instead they need to be downgraded. Remember - b(ionic) comes before x(enial):
root@satellite:~# apt-get install libboost-context1.67.0-icinga=1.67.0-13.1.bionic libboost-coroutine1.67.0-icinga=1.67.0-13.1.bionic libboost-program-options1.67.0-icinga=1.67.0-13.1.bionic libboost-regex1.67.0-icinga=1.67.0-13.1.bionic libboost-system1.67.0-icinga=1.67.0-13.1.bionic libboost-thread1.67.0-icinga=1.67.0-13.1.bionic
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be DOWNGRADED:
libboost-context1.67.0-icinga libboost-coroutine1.67.0-icinga libboost-program-options1.67.0-icinga libboost-regex1.67.0-icinga libboost-system1.67.0-icinga libboost-thread1.67.0-icinga
0 upgraded, 0 newly installed, 6 downgraded, 0 to remove and 0 not upgraded.
3 not fully installed or removed.
Need to get 1,805 kB of archives.
After this operation, 16.4 kB disk space will be freed.
Do you want to continue? [Y/n] y
Get:1 http://packages.icinga.com/ubuntu icinga-bionic/main amd64 libboost-context1.67.0-icinga amd64 1.67.0-13.1.bionic [228 kB]
Get:2 http://packages.icinga.com/ubuntu icinga-bionic/main amd64 libboost-system1.67.0-icinga amd64 1.67.0-13.1.bionic [229 kB]
Get:3 http://packages.icinga.com/ubuntu icinga-bionic/main amd64 libboost-thread1.67.0-icinga amd64 1.67.0-13.1.bionic [267 kB]
Get:4 http://packages.icinga.com/ubuntu icinga-bionic/main amd64 libboost-coroutine1.67.0-icinga amd64 1.67.0-13.1.bionic [243 kB]
Get:5 http://packages.icinga.com/ubuntu icinga-bionic/main amd64 libboost-program-options1.67.0-icinga amd64 1.67.0-13.1.bionic [356 kB]
Get:6 http://packages.icinga.com/ubuntu icinga-bionic/main amd64 libboost-regex1.67.0-icinga amd64 1.67.0-13.1.bionic [482 kB]
Fetched 1,805 kB in 0s (11.0 MB/s)
dpkg: warning: downgrading libboost-context1.67.0-icinga:amd64 from 1.67.0-13.1.xenial to 1.67.0-13.1.bionic
(Reading database ... 224253 files and directories currently installed.)
Preparing to unpack .../0-libboost-context1.67.0-icinga_1.67.0-13.1.bionic_amd64.deb ...
Unpacking libboost-context1.67.0-icinga:amd64 (1.67.0-13.1.bionic) over (1.67.0-13.1.xenial) ...
dpkg: warning: downgrading libboost-system1.67.0-icinga:amd64 from 1.67.0-13.1.xenial to 1.67.0-13.1.bionic
Preparing to unpack .../1-libboost-system1.67.0-icinga_1.67.0-13.1.bionic_amd64.deb ...
Unpacking libboost-system1.67.0-icinga:amd64 (1.67.0-13.1.bionic) over (1.67.0-13.1.xenial) ...
dpkg: warning: downgrading libboost-thread1.67.0-icinga:amd64 from 1.67.0-13.1.xenial to 1.67.0-13.1.bionic
Preparing to unpack .../2-libboost-thread1.67.0-icinga_1.67.0-13.1.bionic_amd64.deb ...
Unpacking libboost-thread1.67.0-icinga:amd64 (1.67.0-13.1.bionic) over (1.67.0-13.1.xenial) ...
dpkg: warning: downgrading libboost-coroutine1.67.0-icinga:amd64 from 1.67.0-13.1.xenial to 1.67.0-13.1.bionic
Preparing to unpack .../3-libboost-coroutine1.67.0-icinga_1.67.0-13.1.bionic_amd64.deb ...
Unpacking libboost-coroutine1.67.0-icinga:amd64 (1.67.0-13.1.bionic) over (1.67.0-13.1.xenial) ...
dpkg: warning: downgrading libboost-program-options1.67.0-icinga:amd64 from 1.67.0-13.1.xenial to 1.67.0-13.1.bionic
Preparing to unpack .../4-libboost-program-options1.67.0-icinga_1.67.0-13.1.bionic_amd64.deb ...
Unpacking libboost-program-options1.67.0-icinga:amd64 (1.67.0-13.1.bionic) over (1.67.0-13.1.xenial) ...
dpkg: warning: downgrading libboost-regex1.67.0-icinga:amd64 from 1.67.0-13.1.xenial to 1.67.0-13.1.bionic
Preparing to unpack .../5-libboost-regex1.67.0-icinga_1.67.0-13.1.bionic_amd64.deb ...
Unpacking libboost-regex1.67.0-icinga:amd64 (1.67.0-13.1.bionic) over (1.67.0-13.1.xenial) ...
Setting up icinga2-common (2.12.4-1.bionic) ...
Setting up libboost-context1.67.0-icinga:amd64 (1.67.0-13.1.bionic) ...
Setting up libboost-program-options1.67.0-icinga:amd64 (1.67.0-13.1.bionic) ...
Setting up libboost-regex1.67.0-icinga:amd64 (1.67.0-13.1.bionic) ...
Setting up libboost-system1.67.0-icinga:amd64 (1.67.0-13.1.bionic) ...
Setting up libboost-thread1.67.0-icinga:amd64 (1.67.0-13.1.bionic) ...
Setting up libboost-coroutine1.67.0-icinga:amd64 (1.67.0-13.1.bionic) ...
Setting up icinga2-bin (2.12.4-1.bionic) ...
Setting up icinga2 (2.12.4-1.bionic) ...
Processing triggers for libc-bin (2.27-3ubuntu1.4) ...
W: APT had planned for dpkg to do more than it reported back (36 vs 40).
Affected packages: icinga2-common:amd64
After this the previous apt-get upgrade was launched again to see if there are still problems:
root@satellite:~# apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
And Icinga 2 is running again:
root@satellite:~# systemctl status icinga2
- icinga2.service - Icinga host/service/network monitoring system
Loaded: loaded (/lib/systemd/system/icinga2.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/icinga2.service.d
|-limits.conf
Active: active (running) since Fri 2021-05-28 11:39:12 CEST; 40s ago
Process: 7783 ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/default/icinga2 (code=exited, status=0/SUCCESS)
Main PID: 7799 (icinga2)
Unfortunately the issue is nothing new and has been reported as an issue. For Ubuntu users, the problem shouldn't happen for a while, as Ubuntu just restarted at the beginning of the alphabet. But Debian users will have the same problem again when upgrading from Buster to Bullseye due to the alphabetical order of the Icinga packages. The definitive solution will be when Icinga's package contain the distribution release version number, not the code name.