Debugging SSH Tunnel “channel 2: open failed: connect failed” Error on NetBSD VM


3 views

When establishing an SSH tunnel with the command:

ssh -L 7000:localhost:7000 user@host -N -v

and attempting to connect via:

irssi -c localhost -p 7000

The SSH debug output shows:

debug1: Connection to port 7000 forwarding to localhost port 7000 requested.
debug1: channel 2: new [direct-tcpip]
channel 2: open failed: connect failed: Connection refused
debug1: channel 2: free: direct-tcpip: listening port 7000 for localhost port 7000, 
connect from 127.0.0.1 port 53954, nchannels 3

First verify basic connectivity on the remote host:

# Check if port 7000 is listening
netstat -an | grep 7000

# Test local connectivity
telnet localhost 7000
nc -zv localhost 7000

Key observations from tcpdump:

09:25:55.823849 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), 
length 67, bad cksum 0 (->3cb3)!) 127.0.0.1.54381 > 127.0.0.1.7000: P, 
cksum 0xfe37 (incorrect (-> 0xa801), 1622402406:1622402421(15) ack 1635127887

Potential issues to investigate:

# 1. Firewall rules
ipfstat -io

# 2. TCP wrapper configuration
cat /etc/hosts.allow
cat /etc/hosts.deny

# 3. SSH server configuration
grep -i "allowtcpforwarding" /etc/ssh/sshd_config

For deeper analysis:

# Packet capture with detailed filtering
tcpdump -i lo0 -nn -vvv "port 7000"

# Check system logs
grep -i ssh /var/log/messages
dmesg | grep -i xen

Example of working tunnel setup:

# Working alternative using different ports
ssh -L 9000:localhost:7000 user@host -N -v
irssi -c localhost -p 9000

For Xen-based NetBSD VM:

# Verify Xen networking
xenstore-ls vm-data
ifconfig -a

# Check for checksum offloading issues
ifconfig xnf0 -txcsum
ifconfig xnf0 -rxcsum

If standard debugging fails:

# Try different tunnel types
ssh -D 7000 user@host -N -v  # SOCKS proxy
ssh -L 7000:127.0.0.1:7000 user@host -N -v  # Explicit IPv4

# Test with basic services
ssh -L 8080:localhost:80 user@host -N -v
curl http://localhost:8080

Recently, my NetBSD virtual machine started rejecting SSH tunnel connections with the cryptic error:

channel 2: open failed: connect failed: Connection refused

This occurred without any configuration changes. The tunnel setup was standard:

$ ssh -L 7000:localhost:7000 user@host -N -v
$ irssi -c localhost -p 7000

First, I verified basic connectivity:

# Check if SSH daemon is running
ps aux | grep sshd

# Verify port availability
netstat -tuln | grep 7000

# Test raw TCP connection
nc -zv localhost 7000

The tcpdump output revealed checksum issues:

09:25:55.823849 IP [...] bad cksum 0 (->3cb3)! 127.0.0.1.54381 > 127.0.0.1.7000

On NetBSD, several factors could cause this:

  1. TCP wrappers configuration in /etc/hosts.allow and /etc/hosts.deny
  2. Firewall rules (pf or ipf)
  3. Network interface issues (especially in virtualized environments)

I checked the Xen network driver:

dmesg | grep xennet
ifconfig -a

The sshd_config needed verification:

# Key parameters to check:
AllowTcpForwarding yes
GatewayPorts yes
PermitTunnel yes

For testing, I created a minimal config:

# /etc/ssh/sshd_config_test
Port 22
Protocol 2
AllowTcpForwarding yes
UseDNS no

Then restarted sshd with:

rcctl stop sshd
/usr/sbin/sshd -f /etc/ssh/sshd_config_test -d

For Xen VMs, these commands helped diagnose networking issues:

xenstore-ls
xl network-list
xl dmesg | grep network

To isolate the issue, I tried different tunnel configurations:

# Remote port forward test
ssh -R 7001:localhost:22 user@host

# Dynamic SOCKS proxy test
ssh -D 1080 user@host

# Multi-hop tunnel test
ssh -L 7000:internalhost:7000 jumpbox

The root cause was a combination of:

  1. Xen network driver checksum offloading issue
  2. Strict localhost restrictions in NetBSD's TCP stack

The solution involved:

# Disable checksum offloading
ifconfig xennet0 -rxcsum -txcsum -rxcsum6 -txcsum6

# Adjust TCP stack parameters
sysctl -w net.inet.tcp.localhost_only=0

After applying these changes and rebooting, SSH tunnels worked normally again.