After upgrading OpenSSH from version 4.3 to 5.8 on my Linux server, I encountered a peculiar issue with PuTTY (win32). While authentication succeeds (password accepted), the session immediately terminates with the fatal error: Server refused to start a shell/command. Interestingly, the same credentials work perfectly when using OpenSSH from Cygwin or native Linux clients.

The root cause appears to be a protocol compatibility issue between PuTTY and the newer OpenSSH server. Let's examine the SSH negotiation process:

# Enable debug logging in sshd_config
LogLevel DEBUG3

# Sample debug output showing the rejection:
debug1: PAM: establishing credentials
debug3: mm_answer_authpassword: sending result 1
debug1: PAM: establishing credentials
debug3: mm_request_receive_expect: received request 122
debug3: mm_request_send: sending request 122
debug3: mm_answer_pwnamallow: sending result 1
debug1: PAM: establishing credentials
debug1: permanently_set_uid: 1000/1000
debug1: SELinux support disabled
debug3: session_pty_req: session 0 alloc /dev/pts/1
debug1: session_pty_req: session 0 alloc /dev/pts/1
debug1: Server refused to start a shell/command

Three potential workarounds exist:

Option 1: Modify sshd_config

Add these directives to /etc/ssh/sshd_config:

# Force legacy behavior
Protocol 2
UsePAM yes
UsePrivilegeSeparation no
Subsystem sftp /usr/lib/openssh/sftp-server

Option 2: PuTTY Configuration Adjustment

In PuTTY's connection settings:

1. Navigate to Connection > SSH > Auth
2. Enable "Allow agent forwarding"
3. In Connection > Data, set "Auto-login username"

Option 3: Alternative Clients

If compatibility persists, consider these modern alternatives:

• Windows Terminal + OpenSSH for Windows
• MobaXterm (handles newer OpenSSH implementations better)
• WSL2 with native Linux SSH client

After applying changes, test with:

ssh -vvv user@server
# Compare output between working/non-working clients

For enterprise environments, consider maintaining backward compatibility through SSH jump hosts running older OpenSSH versions as intermediaries.

---

After upgrading OpenSSH from version 4.3 to 5.8 on my server, PuTTY clients started failing with the error message Server refused to start a shell/command, while native OpenSSH clients (like Cygwin's implementation) continued working normally. This indicates a compatibility issue between PuTTY and the newer OpenSSH server configuration.

The root cause lies in OpenSSH 5.8's stricter default security settings. Specifically:

• New default PermitUserEnvironment setting (now no)
• More restrictive AcceptEnv behavior
• Changed handling of SSH protocol extensions

First, let's examine the SSH handshake by running PuTTY with debug logging:

putty.exe -v -ssh user@hostname -pw password -log debug.log

Typical problematic output shows:

debug1: Sending environment.
debug1: Sending env LC_CTYPE = en_US.UTF-8
debug1: Sending env LC_COLLATE = en_US.UTF-8
debug1: Remote: Server refused to start a shell/command.

Modify /etc/ssh/sshd_config with these critical settings:

AcceptEnv LANG LC_*
PermitUserEnvironment yes
UsePAM yes

Then restart the SSH service:

sudo service ssh restart
# Or for systemd systems:
sudo systemctl restart sshd

For PuTTY-specific solutions:

1. Disable all environment variable sending in PuTTY configuration:
   • Connection > SSH > Auth > "Attempt authentication using Pageant" (uncheck)
   • Connection > Data > "Environment variables" (leave empty)
2. Alternative connection method using plink:

plink.exe -ssh -batch -no-antispoof user@hostname command

For stubborn cases, force protocol version 2 with specific cipher:

putty.exe -2 -cipher aes256-cbc user@hostname

Or in PuTTY configuration:

• Connection > SSH > "Protocol version" (select 2 only)
• Connection > SSH > Cipher (select "aes256-cbc")

After applying fixes, verify with:

ssh -v user@hostname echo "Test successful"
# Expected output:
debug1: Sending command: echo "Test successful"
Test successful