Excessive 404 errors from malicious bots can strain server resources and clutter logs. Fail2ban offers a powerful way to automatically block IPs exhibiting this behavior. Here's how to implement a custom solution.

First, create a new filter file for Apache 404 errors:

sudo nano /etc/fail2ban/filter.d/apache-404.conf

Add this regex pattern to detect consecutive 404 errors:

[Definition]
failregex = ^<HOST>.*"(GET|POST).*" 404
ignoreregex = .*(googlebot|bingbot|slurp).*

Edit your jail.local file:

sudo nano /etc/fail2ban/jail.local

Add this configuration:

[apache-404]
enabled = true
port = http,https
filter = apache-404
logpath = /var/log/apache2/access.log
maxretry = 3
findtime = 600
bantime = 86400

The ignoreregex in our filter excludes major crawlers. For additional verification, you could implement a more sophisticated check:

ignoreregex = .*(googlebot|bingbot|slurp|yandex|duckduckbot).*|.*HTTP/1.\d" 404 .*(\.jpg|\.png|\.css|\.js)

Always test new filters before applying them:

sudo fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-404.conf

For better performance, consider combining this with other Apache rules:

[apache-badbots]
enabled = true

Remember to restart Fail2ban after changes:

sudo systemctl restart fail2ban

Check your bans with:

sudo fail2ban-client status apache-404

And verify they appear in your iptables:

sudo iptables -L -n

---

Many web servers face constant probes from bots scanning for vulnerabilities or abandoned resources. These requests generate excessive 404 errors, wasting server resources and potentially masking real issues in your logs. Fail2ban provides an elegant solution by automatically banning IPs exhibiting this behavior.

First, let's create a custom filter for detecting 404 patterns. Create a new file at /etc/fail2ban/filter.d/apache-404.conf:

[Definition]
failregex = ^<HOST>.*"(GET|POST|HEAD).*" (404|444|403).*$
ignoreregex = .*(googlebot|bingbot|yandex|duckduckbot).*

Add this to your /etc/fail2ban/jail.local:

[apache-404]
enabled  = true
port     = http,https
filter   = apache-404
logpath  = /var/log/apache2/access.log
maxretry = 3
findtime = 600
bantime  = 86400

For more granular control, consider these additional parameters:

# Whitelist trusted crawlers
ignoreip = 192.168.1.0/24 66.249.64.0/19

# Adjust banning duration
bantime.increment = true
bantime.factor = 2
bantime.maxtime = 604800

Test your filter with:

fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-404.conf

Here's what a typical ban entry looks like in /var/log/fail2ban.log:

2023-03-15 14:22:18,420 fail2ban.actions [1234]: NOTICE [apache-404] Ban 185.143.223.67

Check active bans with:

fail2ban-client status apache-404

To unban an IP manually:

fail2ban-client set apache-404 unbanip 192.168.1.100