Of course we cannot always share details about our work with customers, but nevertheless it is nice to show our technical achievements and share some of our implemented solutions.
When using an upstream using proxy_pass withing a Nginx location, it (mostly) works out of the box. But as the Internet (and it's security settings) is becoming more complex, unexpected SSL errors could now show up.
This article covers the SSL alert number 40, which could show up when the upstream server's TLS configuration is unable to handle the requested domain.
Note: Looking for SSL alert number 47? See Nginx reverse proxy error: SSL alert number 47 while SSL handshaking to upstream.
In Nginx, a specific location was defined which should load the content from an (external) upstream:
location = /uripath {
proxy_pass https://external.example.com/;
}
When this location was now accessed using a browser or curl, a 502 error would be returned from Nginx. A closer look into the debug error logs from this domain would show that there was a SSL handshake error with SSL alert number 40 causing the upstream server to be unavailable:
2021/08/30 11:18:32 [error] 2214955#2214955: *231 SSL_do_handshake() failed (SSL: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:SSL alert number 40) while SSL handshaking to upstream, client: 127.0.0.1, server: www.example.com, request: "GET /uripath HTTP/2.0", upstream: "https://104.26.3.5:443/", host: "www.example.com"
2021/08/30 11:18:32 [debug] 2214955#2214955: *231 http upstream ssl handshake: "/uripath?"
2021/08/30 11:18:32 [debug] 2214955#2214955: *231 http next upstream, 2
2021/08/30 11:18:32 [debug] 2214955#2214955: *231 free rr peer 6 4
2021/08/30 11:18:32 [warn] 2214955#2214955: *231 upstream server temporarily disabled while SSL handshaking to upstream, client: 127.0.0.1, server: www.example.com, request: "GET /uripath TP/2.0", upstream: "https://104.26.3.5:443/", host: "www.example.com"
The requested upstream domain (external.example.com) was resolved into an IP address (here: Cloudflare) but the connection to it failed. Hence Nginx disabled the upstream server -> with the result of a 502 error presented to the client (127.0.0.1).
Luckily openssl can be used to reproduce the error and obtain more information.
By simply connecting to the upstream's server IP address, the same SSL error is showing up:
$ openssl s_client -connect 104.26.3.5:443
CONNECTED(00000003)
140542033261888:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1543:SSL alert number 40
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 283 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
However when adding the domain name of the upstream server, the TLS connection can be established:
$ openssl s_client -connect 104.26.3.5:443 -servername external.example.com
CONNECTED(00000003)
depth=2 C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
verify return:1
depth=1 C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
verify return:1
depth=0 C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = sni.cloudflaressl.com
verify return:1
---
Certificate chain
0 s:C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = sni.cloudflaressl.com
i:C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
1 s:C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
i:C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
---
Server certificate
-----BEGIN CERTIFICATE-----
[...]
This means that there is no default certificate installed on the upstream server and a servername (SNI) must be used to successfully establish a connection.
To tell Nginx that the (original) server name (the upstream domain itself) should be used, a specific parameter proxy_ssl_server_name can be added:
location = /uripath {
proxy_ssl_server_name on;
proxy_pass https://external.example.com/;
}
After a Nginx reload, the request from the client now works and the connection to the upstream server can now be established. A look into the Nginx debug log shows that a specific SSL server name was added in the upstream request:
2021/08/30 11:50:58 [debug] 2240173#2240173: *77 http upstream request: "/uripath?"
2021/08/30 11:50:58 [debug] 2240173#2240173: *77 http upstream send request handler
2021/08/30 11:50:58 [debug] 2240173#2240173: *77 malloc: 0000558101039030:88
2021/08/30 11:50:58 [debug] 2240173#2240173: *77 upstream SSL server name: "external.example.com"
2021/08/30 11:50:58 [debug] 2240173#2240173: *77 set session: 0000000000000000
2021/08/30 11:50:58 [debug] 2240173#2240173: *77 tcp_nodelay
2021/08/30 11:50:58 [debug] 2240173#2240173: *77 SSL_do_handshake: -1
2021/08/30 11:50:58 [debug] 2240173#2240173: *77 SSL_get_error: 2
2021/08/30 11:50:58 [debug] 2240173#2240173: *77 SSL handshake handler: 0
2021/08/30 11:50:58 [debug] 2240173#2240173: *77 SSL_do_handshake: -1
2021/08/30 11:50:58 [debug] 2240173#2240173: *77 SSL_get_error: 2
2021/08/30 11:50:58 [debug] 2240173#2240173: *77 SSL handshake handler: 1
2021/08/30 11:50:58 [debug] 2240173#2240173: *77 SSL_do_handshake: -1
2021/08/30 11:50:58 [debug] 2240173#2240173: *77 SSL_get_error: 2
2021/08/30 11:50:58 [debug] 2240173#2240173: *77 SSL handshake handler: 0
2021/08/30 11:50:58 [debug] 2240173#2240173: *77 save session: 00005581012175D0
2021/08/30 11:50:58 [debug] 2240173#2240173: *77 SSL_do_handshake: 1
2021/08/30 11:50:58 [debug] 2240173#2240173: *77 SSL: TLSv1.2, cipher: "ECDHE-ECDSA-CHACHA20-POLY1305 TLSv1.2 Kx=ECDH Au=ECDSA Enc=CHACHA20/POLY1305(256) Mac=AEAD"
2021/08/30 11:50:58 [debug] 2240173#2240173: *77 http upstream ssl handshake: "/uripath?"
2021/08/30 11:50:58 [debug] 2240173#2240173: *77 http upstream send request
2021/08/30 11:50:58 [debug] 2240173#2240173: *77 http upstream send request body