How to Recover SSH Access After iptables Misconfiguration Lockout


2 views

We've all been there - making firewall changes that seemed logical at the time, only to find ourselves completely locked out of our own server. Here's a detailed guide on recovering SSH access when iptables rules go wrong, based on real troubleshooting experience.

The core issue typically occurs when:

  1. New rules are added in the wrong order
  2. The REJECT/DROP all rule moves above service rules
  3. Rules aren't properly saved before testing

In this specific case, the admin:

1. Added FTP rules (20/21)
2. Moved REJECT all below the new rules
3. Restarted iptables
4. Later found SSH (22) and HTTP (80) blocked

Method 1: Hosting Provider Console Access

If available, use the provider's web console or serial console access:

# Once logged in via console:
service iptables stop
# Or for newer systems:
systemctl stop iptables

Method 2: Temporary Rule via crontab (If Already Set Up)

For servers with existing cron jobs that might still execute:

# Add to crontab -e via console
* * * * * /sbin/iptables -I INPUT -p tcp --dport 22 -j ACCEPT
* * * * * /sbin/service iptables save

Method 3: Editing Rules Through Alternative Services

If web services are still running and you have PHP/Perl access:

<?php
// emergency.php
exec('/sbin/iptables -I INPUT -p tcp --dport 22 -j ACCEPT');
exec('/sbin/service iptables save');
?>

For reference, here's a safer default iptables configuration:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
# Allow established connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# Allow localhost
-A INPUT -i lo -j ACCEPT
# SSH
-A INPUT -p tcp --dport 22 -j ACCEPT
# HTTP/HTTPS
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT
# FTP
-A INPUT -p tcp --dport 20:21 -j ACCEPT
# Ping
-A INPUT -p icmp -j ACCEPT
# Log denied
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7
# Reject all other
-A INPUT -j REJECT --reject-with icmp-port-unreachable
COMMIT

Essential safety measures:

  • Use iptables-apply for testing rules
  • Implement a cron-based firewall reset as backup
  • Maintain console access as last resort
  • Consider using fail2ban instead of raw iptables for SSH protection
# Example safety cron job
@reboot /sbin/iptables -F
@reboot /sbin/iptables -A INPUT -p tcp --dport 22 -j ACCEPT

We've all been there - making firewall changes that seemed logical in the moment, only to find ourselves completely locked out of our own server. The specific case where SSH access gets blocked after modifying iptables rules is particularly common (and painful) when:

  • Adding new service ports while forgetting about existing connections
  • Moving the REJECT/DROP rule above service allowances
  • Failing to test rules before saving permanent configurations

In this CentOS 5.6 scenario, the critical mistake was restructuring the rule order without considering the default policies. The sequence:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -p tcp --dport 22 -j ACCEPT
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-port-unreachable  # Problematic early rejection
-A INPUT -p tcp --dport 20 -j ACCEPT   # Added FTP rules
-A INPUT -p tcp --dport 21 -j ACCEPT
COMMIT

When physical access isn't available, try these methods in order:

1. Web-based Console Access

Most hosting providers offer emergency console access:

# Once logged in via console:
service iptables stop
# Or for CentOS 6+:
systemctl stop firewalld

2. Temporary Rule Injection via Cron

If you had a cron job scheduled before the lockout:

# Add to crontab -e
* * * * * /sbin/iptables -I INPUT 1 -p tcp --dport 22 -j ACCEPT

3. Kernel Command Line (Requires Reboot)

For providers supporting kernel parameter editing:

linux /vmlinuz-3.10.0-1160.el7.x86_64 root=/dev/mapper/centos-root ip=dhcp \
  enforcing=0 iptables=0

The correct approach for maintaining SSH access while adding services:

*filter
:INPUT DROP [0:0]   # Default drop policy
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]

# Essential system services
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

# Service ports (order doesn't matter with default DROP)
-A INPUT -p tcp --dport 22 -j ACCEPT  # SSH
-A INPUT -p tcp --dport 80 -j ACCEPT  # HTTP
-A INPUT -p tcp --dport 20 -j ACCEPT  # FTP
-A INPUT -p tcp --dport 21 -j ACCEPT

# Optional: Rate limiting for SSH
-A INPUT -p tcp --dport 22 -m state --state NEW -m recent --set
-A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 -j DROP

Always test new rules before applying them permanently:

# Test sequence:
iptables -F
iptables -P INPUT ACCEPT
# Add temporary rules
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
...
# Verify connectivity
telnet localhost 22
# If working, save permanent config
service iptables save

For modern systems using firewalld (CentOS/RHEL 7+):

firewall-cmd --permanent --add-service=ssh
firewall-cmd --permanent --add-service=ftp
firewall-cmd --permanent --add-service=http
firewall-cmd --reload