Troubleshooting SSH Reverse Port Forwarding: Why Bind Fails on Public IP Address

When setting up SSH reverse port forwarding, you'd expect the remote server to bind to its public IP address by default. However, my recent server migration revealed this isn't always the case. Here's what I discovered when my trusty autossh script suddenly stopped working:

Original command:
sudo autossh -M 5114 -D 7474 -R servers_public_ip:2695:localhost:22 -i /home/user/.ssh/id_rsa2 remote@servers_public_ip

The lsof output showed some interesting behavior:

sshd      27570   remote    3u  IPv4 161761      0t0  TCP servers_public_ip:22->67-213-103-227.eastlink.ca:62812 (ESTABLISHED)
sshd      27570   remote    7u  IPv6 161773      0t0  TCP ip6-localhost:5114 (LISTEN)
sshd      27570   remote    8u  IPv4 161774      0t0  TCP localhost:5114 (LISTEN)
sshd      27570   remote    9u  IPv6 161777      0t0  TCP ip6-localhost:2695 (LISTEN)
sshd      27570   remote   10u  IPv4 161778      0t0  TCP localhost:2695 (LISTEN)

After extensive testing, I found the culprit in the SSH server configuration. The key parameter is:

# In /etc/ssh/sshd_config
GatewayPorts yes

Without this setting, SSH will only bind reverse forwarded ports to loopback addresses (localhost), not to the public interface. This explains why my connections were only showing up as localhost bindings.

Here's the full procedure to make it work:

Edit the SSH server config:
```
sudo nano /etc/ssh/sshd_config
```
Add or modify:
```
AllowTcpForwarding yes
GatewayPorts yes
```
Restart SSH:
```
sudo systemctl restart sshd
```

Run the autossh command with proper binding:

sudo autossh -M 5114 -N -R *:2695:localhost:22 -i /home/user/.ssh/id_rsa2 remote@servers_public_ip

If you can't modify sshd_config, you can use socat as a workaround:

# On the remote server after establishing SSH connection:
socat TCP-LISTEN:2695,bind=servers_public_ip,fork TCP:localhost:2695

Or configure iptables to forward the port:

sudo iptables -t nat -A PREROUTING -p tcp --dport 2695 -j DNAT --to-destination 127.0.0.1:2695
sudo iptables -A FORWARD -p tcp -d 127.0.0.1 --dport 2695 -j ACCEPT

After implementing the solution, verify with:

netstat -tulnp | grep 2695
# Should show the public IP in the Local Address column

Test the connection from another machine:

ssh -p 2695 user@servers_public_ip

When running the following autossh command for reverse tunneling:

sudo autossh -M 5114 -D 7474 -R servers_public_ip:2695:localhost:22 -i /home/user/.ssh/id_rsa2 remote@servers_public_ip

The lsof output reveals the port is only binding to localhost:

sshd      27570   remote    9u  IPv6 161777      0t0  TCP ip6-localhost:2695 (LISTEN)
sshd      27570   remote   10u  IPv4 161778      0t0  TCP localhost:2695 (LISTEN)

For proper public IP binding, these elements must be configured:

# In /etc/ssh/sshd_config
GatewayPorts yes
AllowTcpForwarding yes
TCPKeepAlive yes

Additionally, verify your server's firewall rules:

sudo iptables -L -n -v | grep 2695
sudo ufw status verbose

1. Test with verbose SSH output:

ssh -vvv -R *:2695:localhost:22 user@servers_public_ip

2. Check system logs for errors:

journalctl -u ssh --since "1 hour ago" | grep -i "bind"

3. Verify if the port is actually listening externally:

nc -zv servers_public_ip 2695

If the issue persists, consider using socat as a fallback:

# On local machine
ssh -L 2695:localhost:2695 user@servers_public_ip

# On remote server (after SSH connection)
socat TCP-LISTEN:2695,bind=servers_public_ip,reuseaddr,fork TCP:localhost:2695

To confirm the SSH daemon's binding behavior:

ss -tulnp | grep 2695
netstat -tulnp | grep sshd

The expected output should show the public IP, not just localhost.