Of course we cannot always share details about our work with customers, but nevertheless it is nice to show our technical achievements and share some of our implemented solutions.
A few days ago it happened: Suddenly we got a storm of alerts on a couple of websites we host, and shortly after that we saw increasing load on the affected web server.
As you can see in the CPU Load graph, the load spiked up sharply. But what would cause that? A huge traffic or visitor spike? Checking the network graph (using check_netio), didn't show that much of an increase:
Once we identified the website (a web shop) causing the load spikes, we first suspected a bug in the web application (Magento). This Magento shop runs behind a Nginx reverse proxy and uses Apache with mod_php. We straced Apache processes and saw loops in calling the same files of the Magento cache over and over again (but this turned out to be just a "correct" behavior because the same sites were opened over and over again - at this point in time we didn't know that yet).
Then we analyzed the Nginx access log files with goaccess and compared the data with the previous days and saw a jump in visits:
By just looking at that graph (note: the screenshot of the graph was obviously taken once the system normalized), something very interesting shows up: The hits (blue) didn't grow in the same way the visits (red) did. If the shop would get a lot of legitimate visitor traffic, both lines would more or less grow in parallel. A quick look at some lines in the access log confirmed: This is a DDOS attack.
Lesson one when you're under attack: Do not panic. Whatever you do, think clearly, make decisions based on facts and information. The DDOS attack won't probably go away in the next few hours anyway, so there's no need to rush to a (bad) decision.
Blocking the attacking IP's was literally impossible, as hundreds of thousands of IP addresses were used to attack. Probably a whole botnet just having fun... However using the goaccess summary of the web logs revealed something interesting: Almost all (97%) of the visits (red) originated from China:
Again: Note the correlation between visits (red) and hits (blue). Europe and North America seemed to have real visitors, compared to the Asia graph.
Now that we knew the origin of the attack, we decided to completely block China using Geo Localization.
As mentioned before, the attacked web shop already used Nginx as a reverse proxy in front of Apache. Using the Nginx ngx_http_geoip_module, a geoip map can be created and actions based on that map can be created. I already used the geoip module in the past but for a different reason (see Different proxy_pass upstream depending on client ip address in Nginx).
The needed packages (geoip-database) were already installed on the server, so we could immediately start with the Nginx config:
# Block countries
geoip_country /usr/share/GeoIP/GeoIP.dat;
map $geoip_country_code $allowed_country {
default yes;
CN no;
}
A map is created based on the GeoIP country code and saved as variable $allowed_country. The default is set to "yes", if the looked up country code is "CN", the variable is set to "no".
Inside the server {} context, the $allowed_country variable is read and if it matches "no", the request is blocked with an immediate http status 444 (connection closed without response):
if ($allowed_country = no) {
return 444;
}
Note: Yes, I'm not a fan of "if" in Nginx either, but we got this solution running asap.
As soon as we activated the GeoIP blockage, the server load and the web shop's response times normalized again. It took another 99'000 blocked ip addresses until the DDOS attack finally slowed down and stopped after 37 hours and 19 minutes.