Infiniroot Blog: We sometimes write, too.

Of course we cannot always share details about our work with customers, but nevertheless it is nice to show our technical achievements and share some of our implemented solutions.

500 Internal Server Error served by Nginx reverse proxy due to worker_connections are not enough

Published on August 14th 2020

A Nginx reverse proxy suddenly started to return error 500 on all kinds of sites. The response varied between a real http error (500) and a non-working http communication.

$ curl https://app.example.com -v
* Rebuilt URL to: https://app.example.com/
*   Trying xxx.xxx.xxx.xxx...
* Connected to app.example.com (xxx.xxx.xxx.xxx) port 443 (#0)
* found 127 certificates in /etc/ssl/certs/ca-certificates.crt
* found 520 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
*      server certificate verification OK
*      server certificate status verification SKIPPED
*      common name: *.example.com (matched)
*      server certificate expiration date OK
*      server certificate activation date OK
*      certificate public key: RSA
*      certificate version: #3
*      subject: OU=Domain Control Validated,OU=Gandi Standard Wildcard SSL,CN=*.example.com
*      start date: Thu, 24 Oct 2019 00:00:00 GMT
*      expire date: Thu, 09 Dec 2021 23:59:59 GMT
*      issuer: C=FR,ST=Paris,L=Paris,O=Gandi,CN=Gandi Standard SSL CA 2
*      compression: NULL
* ALPN, server accepted to use http/1.1
> GET / HTTP/1.1
> Host: app.example.com
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< Server: nginx
< Date: Fri, 14 Aug 2020 06:16:54 GMT
< Content-Type: text/html
< Content-Length: 10
< Connection: close
< ETag: "5f362c2c-a"
<
Error 50x
* Closing connection 0

$ curl https://app.example.com -v
* Rebuilt URL to: https://app.example.com/
*   Trying xxx.xxx.xxx.xxx...
* Connected to app.example.com (xxx.xxx.xxx.xxx) port 443 (#0)
* found 127 certificates in /etc/ssl/certs/ca-certificates.crt
* found 520 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* gnutls_handshake() failed: The TLS connection was non-properly terminated.
* Closing connection 0
curl: (35) gnutls_handshake() failed: The TLS connection was non-properly terminated.

Error logs to the help

Luckily there are error logs available - and the logged error is very much self-explaining:

2020/08/14 08:15:31 [alert] 22158#22158: *469070133 768 worker_connections are not enough while connecting to upstream, client: 54.93.81.216, server: app.example.com, request: "GET /v3/connect/config HTTP/1.1", upstream: "http://127.0.0.1:8105/v3/connect/config", host: "app.example.com"

Increasing worker_connections to fix

worker_connections is a global config option in the events context. On most Nginx installations this can be found in /etc/nginx/nginx.conf.

events {
worker_connections 768;
        # multi_accept on;
}

The default value, here on an Ubuntu 18.04 system, is set to 768 worker_connections. In this case, this reverse proxy received more connections than this and the default worker_connections limit is not high enough.

After setting worker_connections to a higher value and a Nginx restart, the errors were gone and the reverse proxy served the application again.

root@nginx:~# grep worker_connections /etc/nginx/nginx.conf
    worker_connections 1024;

root@nginx:~# systemctl restart nginx

$ curl https://app.example.com -I
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 14 Aug 2020 06:18:45 GMT
Content-Type: application/json
Connection: keep-alive
Vary: Accept-Encoding
Cache-Control: no-cache, no-store, must-revalidate
Expires: Wed 24 Feb 1982 18:42:00 GMT
X-Frame-Options: deny

Monitoring the worker_connections

There are two ways of monitoring the Nginx reverse proxy and therefore "kind of" monitoring its number of worker connections:

Unfortunately both ways do not show the real Nginx-internal number of concurrent worker_connections. But they can help indicate when the current limits won't be enough to handle communications.

The Nginx Status page can be actively monitored with the monitoring plugin check_nginx_status. The results can be graphed and this is a helpful indicator to see whether or not Nginx will run into problems (or to detect other problems such as DDOS attacks):

Nginx graph of active workers using check_nginx_status

By using the lsof command (netstat would be an alternative), the number of ESTABLISHED tcp connections from and to Nginx listening on ports 80 and 443 are counted:

root@nginx:~# lsof -i :443,80 | grep ESTABLISHED -c
67