Of course we cannot always share details about our work with customers, but nevertheless it is nice to show our technical achievements and share some of our implemented solutions.
Any application with a listener, such as (web) applications, creates a socket for each connection. The number of sockets such an application can create are limited by the so-called "max open files" limit.
In the past (pre Systemd), this problem was solved by adjusting /etc/security/limits.conf. But in the days of applications started by Systemd, this init system applies its own limits. I recommend to read the article "Nginx socket() failed 24: too many open files" to get more details.
HAProxy makes no exception in this case as it is basically also an application with listener(s), creating sockets for each connection. And of course, HAProxy, too, can run into this max open files limit. In such an event, the following log entries can be seen in the HAProxy logs:
Proxy www-out reached process FD limit (maxsock=1023). Please check 'ulimit-n' and restart.
Where "www-out" is the backend name.
The logged error is very helpful actually, because it shows the current limit (1023). This can be verified by checking the limits of the HAProxy process:
root@linux:~# ps auxf|grep haproxy
root 1899651 0.2 0.0 39412 14884 ? Ss 08:03 0:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
haproxy 1899654 7.2 0.0 556420 15060 ? Sl 08:03 0:00 \_ /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
root@linux:~# cat /proc/1899651/limits | grep "open"
Max open files 1023 524288 files
root@linux:~# cat /proc/1899654/limits | grep "open"
Max open files 1023 524288 files
Both master (PID 1899651) and child (PID 1899654) processes have a "max open files" limit of 1023.
As mentioned at the begin, in the past this would have been solved by modifying /etc/security/limits.conf but with Systemd this is handled differently. One way would be to copy the "original" Systemd unit file of the HAProxy service (/lib/systemd/system/haproxy.service) to /etc/systemd/system/haproxy.service and manually add the LimitNOFILE parameter in it. But a future (security) update of HAProxy could adjust the HAProxy service unit and the update would have no effect.
So the better solution is to create a specific Systemd "sub-unit" configuration. This is done by using a directory named after the service. For HAProxy this is: /etc/systemd/system/haproxy.service.d.
root@linux:~# mkdir /etc/systemd/system/haproxy.service.d
Inside this directory any file with a suffix of .conf will be read by Systemd. Options defined here will append or overwrite existing Systemd unit options.
To increase the limit, create the file /etc/systemd/system/haproxy.service.d/limits.conf with the following content:
root@linux:~# cat /etc/systemd/system/haproxy.service.d/limits.conf
[Service]
LimitNOFILE=500000
Now reload Systemd to read the added configs and then restart HAProxy:
root@linux:~# systemctl daemon-reload
root@linux:~# systemctl restart haproxy
The new limits should now be enabled and this can be verified by reading the limits of the new HAProxy processes:
root@linux:~# ps auxf|grep haproxy
root 1902679 1.0 0.1 84364 46036 ? Ss 08:06 0:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
haproxy 1902682 5.3 0.1 614484 44768 ? Sl 08:06 0:00 \_ /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
root@linux:~# cat /proc/1902679/limits |grep "open"
Max open files 499999 500000 files