Comparing DenyHosts vs Fail2Ban vs IPTables: Optimal SSH/FTP Brute Force Protection Strategies for Linux Servers


3 views

Brute force attacks remain one of the most persistent threats to Linux servers, particularly for services like SSH (port 22) and FTP (port 21). Attackers use automated tools to systematically guess credentials, potentially gaining unauthorized access to your system. Let's examine three primary defense mechanisms.

DenyHosts is a Python script that monitors auth logs and blocks suspicious IPs by adding them to /etc/hosts.deny. While simple, it's becoming outdated (last updated in 2015).

# Sample DenyHosts configuration (/etc/denyhosts.conf)
SECURE_LOG = /var/log/auth.log
HOSTS_DENY = /etc/hosts.deny
PURGE_DENY = 1w  # Auto-remove blocked IPs after 1 week
ADMIN_EMAIL = admin@example.com

Fail2Ban scans log files for patterns and updates firewall rules dynamically. It supports multiple services and offers more granular control than DenyHosts.

# Example Fail2Ban jail.local configuration for SSH
[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
findtime = 10m
bantime = 1h

While not a monitoring tool, IPTables can be configured with brute-force protection rules. This provides immediate blocking without log analysis.

# IPTables rule to limit SSH connections
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --set --name SSH
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --rttl --name SSH -j DROP
Feature DenyHosts Fail2Ban IPTables
Configuration Complexity Low Medium High
Service Coverage Limited Extensive Protocol-level
Performance Impact Low Medium Low
Maintenance Status Unmaintained Active Kernel-level

For maximum security, consider combining these tools:

  1. Use IPTables for basic rate limiting
  2. Implement Fail2Ban for intelligent pattern detection
  3. Change default ports (e.g., SSH to 2222)
  4. Implement key-based authentication
# Combined approach example (Fail2Ban + IPTables)
fail2ban-client set sshd addignoreip 192.168.1.0/24
iptables -N FAIL2BAN
iptables -A INPUT -j FAIL2BAN

Regularly check your protection systems:

# Check Fail2Ban status
fail2ban-client status sshd

# View current IPTables rules
iptables -L -n -v

# Monitor DenyHosts blocks
cat /etc/hosts.deny | grep sshd

Brute-force attacks remain one of the most common security threats for LAMP servers. Attackers use automated scripts to try thousands of username/password combinations through SSH, FTP, and other services. According to recent security reports, an unprotected SSH server receives hundreds of brute-force attempts per day.

# Sample IPTables rule to block repeated SSH attempts
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --set --name SSH
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --name SSH -j DROP

DenyHosts monitors authentication logs (/var/log/auth.log) and automatically blocks suspicious IPs by adding them to /etc/hosts.deny. Configuration is done through /etc/denyhosts.conf:

# DenyHosts configuration example
SECURE_LOG = /var/log/auth.log
HOSTS_DENY = /etc/hosts.deny
PURGE_DENY = 1w
BLOCK_SERVICE = sshd
ADMIN_EMAIL = admin@example.com

Fail2Ban offers more granular control through its jail system. A typical SSH protection configuration (/etc/fail2ban/jail.local):

[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 3600
findtime = 600

For those preferring native solutions, IPTables can be configured with recent module for basic protection:

# Comprehensive IPTables protection
iptables -N SSH_PROTECT
iptables -A INPUT -p tcp --dport 22 -j SSH_PROTECT
iptables -A SSH_PROTECT -m recent --name SSH --set
iptables -A SSH_PROTECT -m recent --name SSH --update --seconds 300 --hitcount 5 -j DROP
Feature DenyHosts Fail2Ban IPTables
Language Python Python C (kernel)
Configuration Single config Multiple jails Rule chains
Log Monitoring Yes Yes No
Performance Impact Medium Medium Low
Protocol Support SSH-focused Multi-service All TCP/UDP

For maximum security, combine these tools:

  1. Use Fail2Ban for application-layer protection
  2. Implement IPTables rate-limiting as first defense
  3. Consider port knocking for hidden services
# Port knocking example (iptables)
iptables -N KNOCKING
iptables -A INPUT -j KNOCKING
iptables -A KNOCKING -p tcp --dport 8000 -m recent --name KNOCK1 --set -j DROP
iptables -A KNOCKING -p tcp --dport 8001 -m recent --name KNOCK1 --remove -j DROP
iptables -A KNOCKING -m recent --name KNOCK2 --rcheck -j ACCEPT

For most LAMP servers:

  • Start with Fail2Ban for its flexibility
  • Add IPTables rules for basic rate-limiting
  • Monitor /var/log/fail2ban.log for attack patterns
  • Consider moving SSH to non-standard port