Of course we cannot always share details about our work with customers, but nevertheless it is nice to show our technical achievements and share some of our implemented solutions.
In the past days, Ubuntu's 16.04 LTS release (Xenial) officially become end of life. This forced a lot of users to switch to the newest LTS 20.04 (Focal) this year - or are still doing migrations right now.
The jump from 16.04 to 20.04 also means a version jump of Nginx: From 1.10.0 on Xenial to 1.18.0 in Focal. We've covered the migration of the Nginx configurations in a separate article: Important Nginx configuration changes after ugprading Ubuntu from 16.04 to 20.04.
However there's something else besides the configuration changes, which is very important: The Nginx reload crash.
Once on Ubuntu 20.04 with Nginx 1.18.0, as you are happily modifying or adding configurations and then reloading, you suddenly notice that HTTP is down and all Nginx processes are gone. What? Why?
This can actually be reproduced by simply reloading Nginx twice (or more) in a row:
root@focal:~# ps aux|grep nginx
root 3356067 0.0 0.1 71600 9788 ? Ss 15:46 0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 3356068 0.0 0.1 73012 15348 ? S 15:46 0:00 nginx: worker process
www-data 3356069 0.0 0.1 73012 15348 ? S 15:46 0:00 nginx: worker process
www-data 3356070 0.0 0.1 73012 15348 ? S 15:46 0:00 nginx: worker process
www-data 3356071 0.0 0.1 73012 15348 ? S 15:46 0:00 nginx: worker process
www-data 3356072 0.0 0.1 73012 15348 ? S 15:46 0:00 nginx: worker process
www-data 3356073 0.0 0.1 73012 15348 ? S 15:46 0:00 nginx: worker process
www-data 3356074 0.0 0.1 73012 15348 ? S 15:46 0:00 nginx: worker process
www-data 3356075 0.0 0.1 73012 15348 ? S 15:46 0:00 nginx: worker process
root 3356102 0.0 0.0 8900 676 pts/0 S+ 15:46 0:00 grep --color=auto nginx
root@focal:~# service nginx reload
root@focal:~# ps aux|grep nginx
root 3356067 0.2 0.3 81404 28848 ? Ss 15:46 0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 3356266 0.0 0.2 81644 22608 ? S 15:46 0:00 nginx: worker process
www-data 3356267 0.0 0.2 81644 22608 ? S 15:46 0:00 nginx: worker process
www-data 3356268 0.0 0.2 81644 22608 ? S 15:46 0:00 nginx: worker process
www-data 3356269 0.0 0.2 81644 22608 ? S 15:46 0:00 nginx: worker process
www-data 3356270 0.0 0.2 81644 22608 ? S 15:46 0:00 nginx: worker process
www-data 3356271 0.0 0.2 81644 22608 ? S 15:46 0:00 nginx: worker process
www-data 3356272 0.0 0.2 81644 22608 ? S 15:46 0:00 nginx: worker process
www-data 3356273 0.0 0.2 81644 22608 ? S 15:46 0:00 nginx: worker process
root 3356296 0.0 0.0 17196 7168 ? R 15:46 0:00 /usr/bin/pgrep nginx
root 3356309 0.0 0.0 8900 736 pts/0 S+ 15:46 0:00 grep --color=auto nginx
root@focal:~# service nginx reload
root@focal:~# ps aux|grep nginx
root 3356355 0.0 0.0 8900 736 pts/0 S+ 15:46 0:00 grep --color=auto nginx
root@focal:~#
Note that after the first reload the Nginx processes were still running. After the second reload they were gone.
Additionally to this fact, the following segmentation fault might be logged in /var/log/kern.log right after the second reload:
root@focal:~# tail -f /var/log/kern.log
May 4 15:46:50 focal kernel: [4141537.262674] nginx[3356067]: segfault at 51 ip 00007f6b7d1535c9 sp 00007ffe634334d0 error 4 in libperl.so.5.30.0[7f6b7d0eb000+166000]
May 4 15:46:50 focal kernel: [4141537.262681] Code: 00 0f b6 40 30 49 c1 ed 03 49 29 c5 0f 84 17 01 00 00 48 8b 76 10 48 8b 52 10 4c 8d 3c fe 4c 8d 0c c2 84 c9 0f 84 c7 02 00 00 <49> 83 39 00 0f 85 ad 03 00 00 49 83 c1 08 49 83 ed 01 49 8d 74 1d
From the logged segmentation fault above, the relevant information is that the error happened in libperl.so.5.30.0. This means the Nginx reload triggered something in Perl which returned a (segmentation fault) error and Nginx itself crashed. But what function inside Nginx is using Perl? Actually nothing - however there's a Nginx module for Perl, which is enabled by default:
root@focal:~# ls -l /etc/nginx/modules-enabled/50-mod-http-perl.conf
lrwxrwxrwx 1 root root 53 Mar 4 15:14 /etc/nginx/modules-enabled/50-mod-http-perl.conf -> /usr/share/nginx/modules-available/mod-http-perl.conf
This module (comparable to Apache's mod_perl2) launches Perl in the background and a bug (more on that later in that article) is causing the segmentation error - and the crash.
Most Nginx users actually don't use Perl applications - so for most users it's safe to disable the Perl module:
root@focal:~# rm -f /etc/nginx/modules-enabled/50-mod-http-perl.conf
root@focal:~# systemctl restart nginx
The multiple reload bug is now gone:
root@focal:~# ps auxf|grep nginx
root 3363494 0.0 0.0 8900 736 pts/0 S+ 15:55 0:00 | \_ grep --color=auto nginx
root 3363349 0.0 0.1 67296 8848 ? Ss 15:55 0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 3363350 0.0 0.1 68724 14704 ? S 15:55 0:00 \_ nginx: worker process
www-data 3363351 0.0 0.1 68724 14220 ? S 15:55 0:00 \_ nginx: worker process
www-data 3363352 0.0 0.1 68724 14220 ? S 15:55 0:00 \_ nginx: worker process
www-data 3363353 0.0 0.1 68724 14220 ? S 15:55 0:00 \_ nginx: worker process
www-data 3363354 0.0 0.1 68724 14220 ? S 15:55 0:00 \_ nginx: worker process
www-data 3363355 0.0 0.1 68724 14220 ? S 15:55 0:00 \_ nginx: worker process
www-data 3363356 0.0 0.1 68724 14220 ? S 15:55 0:00 \_ nginx: worker process
www-data 3363357 0.0 0.1 68724 14220 ? S 15:55 0:00 \_ nginx: worker process
root@focal:~# service nginx reload
root@focal:~# ps auxf|grep nginx
root 3363553 0.0 0.0 17196 7036 ? R 15:56 0:00 \_ /usr/bin/pgrep nginx
root 3363555 0.0 0.0 8900 724 pts/0 S+ 15:56 0:00 | \_ grep --color=auto nginx
root 3363349 0.2 0.2 74184 23384 ? Ss 15:55 0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 3363531 0.0 0.2 74424 20028 ? S 15:55 0:00 \_ nginx: worker process
www-data 3363532 0.0 0.2 74424 19648 ? S 15:55 0:00 \_ nginx: worker process
www-data 3363533 0.0 0.2 74424 19648 ? S 15:55 0:00 \_ nginx: worker process
www-data 3363534 0.0 0.2 74424 19648 ? S 15:55 0:00 \_ nginx: worker process
www-data 3363535 0.0 0.2 74424 19648 ? S 15:55 0:00 \_ nginx: worker process
www-data 3363536 0.0 0.2 74424 19648 ? S 15:55 0:00 \_ nginx: worker process
www-data 3363537 0.0 0.2 74424 19648 ? S 15:55 0:00 \_ nginx: worker process
www-data 3363538 0.0 0.2 74424 19648 ? S 15:55 0:00 \_ nginx: worker process
root@focal:~# service nginx reload
root@focal:~# ps auxf|grep nginx
root 3363618 0.0 0.0 8900 740 pts/0 S+ 15:56 0:00 | \_ grep --color=auto nginx
root 3363349 0.4 0.2 74208 23424 ? Ss 15:55 0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 3363586 0.0 0.2 74448 20144 ? S 15:56 0:00 \_ nginx: worker process
www-data 3363587 0.0 0.2 74448 19684 ? S 15:56 0:00 \_ nginx: worker process
www-data 3363588 0.0 0.2 74448 19684 ? S 15:56 0:00 \_ nginx: worker process
www-data 3363589 0.0 0.2 74448 19684 ? S 15:56 0:00 \_ nginx: worker process
www-data 3363590 0.0 0.2 74448 19684 ? S 15:56 0:00 \_ nginx: worker process
www-data 3363591 0.0 0.2 74448 19684 ? S 15:56 0:00 \_ nginx: worker process
www-data 3363592 0.0 0.2 74448 19684 ? S 15:56 0:00 \_ nginx: worker process
www-data 3363593 0.0 0.2 74448 19684 ? S 15:56 0:00 \_ nginx: worker process
The Nginx processes are still running happily after the second reload. The workaround works!
But what is actually causing it? A bug in Nginx? Or in Ubuntu itself (e.g. in the Systemd reload mechanism)? Research leads to the obvious bug in Ubuntu's launchpad: Bug #1897561 describes the exact same problems and was confirmed by more than a dozen other Ubuntu users. However a more revealing bug is Nginx bug #1831. This bug was already reported a while ago, in August 2019, and already affected Nginx 1.16. The initial thought was that the Nginx Perl module itself causes the reload crash, however it turns out to be a bug in Perl itself. The bug is fixed in Perl 5.32 and 5.30.2 - however Ubuntu 20.04 ships with the buggy Perl 5.30.0 version.
root@focal:~# dpkg -l|grep "ii perl"
ii perl 5.30.0-9ubuntu0.2 amd64 Larry Wall's Practical Extraction and Report Language
ii perl-base 5.30.0-9ubuntu0.2 amd64 minimal Perl system
ii perl-modules-5.26 5.26.1-6ubuntu0.5 all Core Perl modules
ii perl-modules-5.30 5.30.0-9ubuntu0.2 all Core Perl modules
So at the end the ball is in the corner of the Ubuntu package maintainers right now. The Perl package needs to be updated to 5.30.2 or at least contain the patch to fix this bug. Until then use the workaround.