Detect and stop 404 attacks with fail2ban

A common penetration test malicious visitors like to deploy is running a script full of not-so-random URIs.

Depending on if they receive a 200, 4xx, or 5xx response, this will let them know certain things about your web app. Such as:

• What tech stack is being used, which can let them know how to exploit the site further
• If files that contain sensitive information are exposed

These types of attacks often result in the generation of a plethora of 404 errors.

"404 attack" isn't the best descriptor of what the attack is, but 404 errors are a byproduct that we can key off of to help stomp these types of attacks.

In addition to the security issues these attacks expose, another reason to consider stomping these attacks is the server load they expend. These attacks will often occur over a relatively short period, but will go through hundreds, or thousands of URLs, which can easily impact the performance of any app hosted on the server.

Set Up Test Server and Site

In this exercise, we'll provision a server using Cleavr and add a WordPress site. Cleavr installs and configures fail2ban, which we'll further configure to detect and squash these 404 attacks.

We'll create a filter rule for fail2ban to check the NGINX access.log to detect if an IP generates too many 404 errors within a specified period of time. If an IP is caught breaking the rules, we'll put them in jail by temporarily banning them from accessing the server.

With a test server ready to go, let's put our detective cap on!

Step 1

SSH into your server. View our guide on how to SSH into your server. Or, use your favorite SFTP client.

Step 2

Add the following new file named nginx-4xx.conf to /etc/fail2ban/filter.d/

[Definition] 
failregex = ^<HOST>.*"(GET|POST).*" (404|444|403|400) .*$ 
ignoreregex =

The above is a filter definition that tells fail2ban to look for errors marked 400, 403, 404, and 444.

Step 3

Now, open /etc/fail2ban/jail.conf and add the following block of code to the end of the file.

[nginx-4xx] 
enabled = true 
port = http,https 
filter = nginx-4xx 
logpath = /var/log/nginx/access.log 
bantime = 1800 
findtime = 10
maxretry = 10

In the above, pay attention to bantime, maxretry, and findtime. This is saying if a user hits the 4xx error 10 times over a 10 second period, then ban the user's IP for 1800 seconds, which is 30 minutes.

Modify the values according to your needs - but, I recommend something that will deter attackers, who tend to probe many pages in less than a minute, but that also will minimally impact legitimate visitors.

Step 4

Now, restart process by running following command:

service fail2ban restart

To view status, including jailed IPs, you can run the following command:

fail2ban-client status nginx-4xx

Step 5 - test it out!

Now, go to the website, type in a URL that doesn't exist and will generate a 404 error, then refresh quickly until you get blocked by the server.

You can then go back to the terminal and run fail2ban-client status nginx-4xx to see that you have been jailed. 🚔

---

It is important to note that this is just one way to handle this type of issue and it may not be the best option for you. You should perform the due diligence and verify this method works for you, your system, and your users.

Comments

[Jose Maria]

Great post! but i think there's something missing in your setup: there's no action to take once the ip address is detected, i think you need to add on step 3 in the jail.com file the line:

action = iptables-allports [name=nginx-4xx] (i wanted to block all ports)

When i tested, the ip address is detected and listed as "banned" but iptables does not have the rule. After adding this action the ip address got effectively blocked.

Regards!

[Melroy van den Berg]

There is a default action in your jail.conf already (jail.local only extends this config further). So actually you should create a jail.local file as well!

Default action in the jail.conf file should be (action_ means just a ban only):
action = %(action_)s

[Melroy van den Berg]

Also don't forget to include HEAD requests you also want to ban those, so it will be: (GET|POST|HEAD)

[Melroy van den Berg]

The filter line is no longer needed in jail.local. See also: the-art-of-web.com/system/fail2ban...