Troubleshooting X11 Forwarding Issues in SSH: Authentication Failures and Connection Rejections


3 views

The fundamental issue manifests when attempting to forward X11 applications through an SSH connection, receiving authentication errors despite proper configuration:

X11 connection rejected because of wrong authentication.
kate: cannot connect to X server localhost:10.0

Let's examine the complete configuration stack that affects X11 forwarding:

Server-side (/etc/ssh/sshd_config)

# Minimum required settings
X11Forwarding yes
X11DisplayOffset 10
X11UseLocalhost yes

# Additional recommended parameters
XAuthLocation /usr/bin/xauth
AddressFamily inet

Client-side (~/.ssh/config)

Host home
    HostName HOME_IP
    User azat
    ForwardX11 yes
    ForwardX11Trusted yes
    # Security recommendation:
    PreferredAuthentications publickey,password

When facing X11 forwarding issues, follow this diagnostic workflow:

1. Verify Basic X11 Functionality

# On local machine:
xhost +
xauth list

# After SSH connection:
echo $DISPLAY
xauth list

2. Check Authentication Protocols

The debug output indicates protocol mismatches:

debug2: X11 connection uses different authentication protocol.
X11 connection rejected because of wrong authentication.

This typically occurs when the remote and local Xauth mechanisms differ. Force a specific protocol:

ssh -o ForwardX11Trusted=yes -Y user@host

Xauthority File Management

The .Xauthority files must contain compatible entries:

# Regenerate Xauth cookies
xauth generate :0 . trusted
xauth add ${HOST}:0 . $(mcookie)

Network-Level Considerations /h2>

For iptables configurations, ensure these ports are open:

iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -p tcp --dport 6010:6019 -j ACCEPT

For KDE environments (as in this case):

# KDE-specific X11 settings
export QT_X11_NO_MITSHM=1
export XAUTHORITY=~/.Xauthority

When standard X11 forwarding fails, consider these alternatives:

1. X2Go Solution

# Install X2Go client
sudo apt install x2goclient

# Configure with:
Protocol: X2Go
Session type: KDE

2. VNC Over SSH

ssh -L 5900:localhost:5900 user@host
vncviewer localhost:0

After applying fixes, test with:

ssh -v -X user@host 'env | grep -E "DISPLAY|XAUTHORITY"; xclock'

When attempting X11 forwarding through SSH, the error X11 connection rejected because of wrong authentication typically indicates a mismatch in X11 authentication protocols or cookie handling between the client and server. From the debug logs, we can see the authentication protocol discrepancy:

debug2: X11 connection uses different authentication protocol.
X11 connection rejected because of wrong authentication.
# Server-side (/etc/ssh/sshd_config):
X11Forwarding yes
X11DisplayOffset 10
X11UseLocalhost no  # Important change from original config

The critical change here is X11UseLocalhost no which allows remote clients to connect to the X11 tunnel.

First verify your X11 forwarding environment variables:

$ ssh -vvv -X user@host
$ echo $DISPLAY
localhost:10.0  # Should match your X11DisplayOffset

Check Xauth entries on both systems:

# On client machine before SSH:
xauth list $DISPLAY

# On server after SSH:
xauth list

The MIT-MAGIC-COOKIE-1 discrepancy suggests we need to ensure consistent authentication methods. Try these steps:

# 1. Generate a new Xauthority file
mv ~/.Xauthority ~/.Xauthority.bak
xauth generate $DISPLAY . untrusted

# 2. Alternatively specify the auth protocol explicitly
ssh -o ForwardX11Trusted=yes -Y user@host

While port 22 is open, X11 forwarding uses additional ports. Check your iptables rules:

# Allow X11 forwarding ports (6000-6063)
iptables -A INPUT -p tcp --dport 6010 -j ACCEPT

For KDE users experiencing this issue with kate, try these additional steps:

# 1. Reset KDE's X11 settings
kwriteconfig5 --file startkderc --group X11 --key Xinerama "false"

# 2. Try with basic X11 apps first
sudo apt install x11-apps
xeyes  # Test basic X11 forwarding

If standard X11 forwarding fails persistently, consider these alternatives:

# 1. Use Xpra for more robust forwarding
xpra start :100
DISPLAY=:100 kate &

# 2. Try VNC as fallback
ssh -L 5901:localhost:5901 user@host