Implementing Persistent Reverse SSH Tunnels for Firewall-Bypassed Linux Maintenance


2 views

When managing Linux machines behind restrictive firewalls, traditional SSH port forwarding becomes impractical. The core issues we face:

  • No firewall modification privileges at remote locations
  • Unpredictable maintenance windows requiring on-demand access
  • Security concerns with permanent open ports

The solution involves creating a persistent outbound connection from the target machine to an accessible jump host:

# On the remote machine (client side)
autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" \
-N -R 2222:localhost:22 jumpuser@your-jump-host.com

This establishes a tunnel where:

  • -M 0 disables monitoring (handled by systemd)
  • ServerAlive packets maintain connection
  • Remote port 2222 forwards to local SSH

For reliable auto-reconnection, create a systemd service:

# /etc/systemd/system/ssh-tunnel.service
[Unit]
Description=Persistent SSH tunnel
After=network.target

[Service]
User=tunneluser
ExecStart=/usr/bin/autossh -M 0 -N \
-o "ServerAliveInterval 30" \
-o "ServerAliveCountMax 3" \
-R 2222:localhost:22 jumpuser@your-jump-host.com
Restart=always
RestartSec=60

[Install]
WantedBy=multi-user.target

After establishing the tunnel, access the remote machine from your jump host:

ssh -p 2222 localuser@localhost
  • Use SSH keys with passphrase protection
  • Implement fail2ban on the jump host
  • Rotate credentials periodically
  • Consider TCP wrappers for additional filtering

Add this cron job to verify tunnel health:

*/5 * * * * pgrep autossh || systemctl restart ssh-tunnel

For modern environments, consider:

# Using Teleport for SSH access
teleport start --roles=node --token=xyz \
--auth-server=teleport.example.com:3080

When managing Linux machines deployed in remote locations with restrictive firewall policies, traditional SSH access becomes problematic. Opening persistent firewall ports creates security vulnerabilities, while manual intervention for temporary access isn't scalable for production environments.

The most reliable approach in constrained environments involves establishing a reverse SSH tunnel. This method works even when you can't modify firewall rules directly. Here's how to implement it:


# On the remote machine (run as cron job or systemd service)
autossh -M 0 -N -R 2222:localhost:22 user@your-public-server.com -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3"

Key components:
- autossh maintains the connection automatically
- -R 2222:localhost:22 forwards remote port 22 to local port 2222
- ServerAlive options prevent stale connections

For completely hands-off operation, create a systemd service:


# /etc/systemd/system/remote-access.service
[Unit]
Description=Persistent reverse SSH tunnel
After=network.target

[Service]
User=autossh
ExecStart=/usr/bin/autossh -M 0 -N -R 2222:localhost:22 user@your-public-server.com -i /home/autossh/.ssh/id_rsa
Restart=always
RestartSec=60

[Install]
WantedBy=multi-user.target

For environments where you can modify firewall rules (via API or configuration), consider this knockd example:


# /etc/knockd.conf
[options]
    UseSyslog

[openSSH]
    sequence    = 7000,8000,9000
    seq_timeout = 10
    command     = /sbin/iptables -A INPUT -s %IP% -p tcp --dport 22 -j ACCEPT
    tcpflags    = syn

[closeSSH]
    sequence    = 9000,8000,7000
    seq_timeout = 10
    command     = /sbin/iptables -D INPUT -s %IP% -p tcp --dport 22 -j ACCEPT
    tcpflags    = syn

Whichever method you choose:

  • Use SSH key authentication exclusively
  • Implement fail2ban on the public-facing server
  • Consider VPN alternatives if managing multiple machines
  • Rotate credentials regularly for reverse tunnel accounts

Common problems and solutions:


# Check tunnel status:
ssh -p 2222 localhost
netstat -tulnp | grep ssh

# Debugging autossh:
journalctl -u remote-access.service -f