Solving SSH Host Key Verification Issues When Using Virtual IP (VIP) Failover in SFTP Environments


2 views

When implementing high-availability solutions with Virtual IPs (VIPs), a common pitfall emerges in SSH/SFTP communications. Consider this production scenario:

Primary:   myservice.mycompany.uk → server1 (SSH host key: KEY_A)
Failover:  myservice.mycompany.uk → server2 (SSH host key: KEY_B)

Client systems caching host keys encounter verification failures during failover events because:

  • StrictHostKeyChecking=yes (security best practice)
  • DNS name remains constant but backend server identity changes

The naive approach of disabling host key checking (StrictHostKeyChecking=no) creates security vulnerabilities. Alternative methods include:

# Bad practice (security risk):
Host *
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null

The duplicate known_hosts entries proposal has merit but faces implementation challenges:

  1. OpenSSH format allows multiple keys per host
  2. PuTTY/Windows registry stores keys differently (no native duplication)

Option 1: Shared Host Keys (Recommended)

Generate identical host keys on both servers:

# On primary server:
ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
# Copy to standby:
scp /etc/ssh/ssh_host_*_key root@server2:/etc/ssh/

Security Note: Rotate shared keys if either server is compromised.

Option 2: Advanced known_hosts Configuration

For OpenSSH clients, implement hashed host entries with multiple keys:

# Generate entry for server1
ssh-keyscan -t rsa server1 >> ~/.ssh/known_hosts

# Append server2's key for same hostname
echo "myservice.mycompany.uk,1.2.3.4 $(cat /etc/ssh/ssh_host_rsa_key.pub)" >> ~/.ssh/known_hosts

Option 3: PuTTY-Specific Implementation

Windows systems require registry modifications:

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\SimonTatham\PuTTY\SshHostKeys]
"rsa2@22:myservice.mycompany.uk"="0x1234...server1_key"
"rsa2@22:myservice.mycompany.uk\\secondary"="0x5678...server2_key"

For dynamic environments, implement key distribution via configuration management:

# Ansible playbook snippet:
- name: Deploy shared SSH host keys
  copy:
    src: "{{ ssh_host_key_dir }}/ssh_host_{{ item }}_key"
    dest: "/etc/ssh/ssh_host_{{ item }}_key"
    mode: 0600
  with_items:
    - rsa
    - ecdsa
    - ed25519

Validate failover behavior using:

#!/bin/bash
# Test host key acceptance
for i in {1..10}; do
    ssh -v -o StrictHostKeyChecking=yes \
        myservice.mycompany.uk "hostname"
    sleep 5
done

When implementing high-availability setups with Virtual IPs (VIPs) for SFTP services, we often encounter SSH host key verification problems. Consider this common scenario:

# Example server configuration
VIP: myservice.mycompany.uk
Primary: server1 (192.168.1.10)
Secondary: server2 (192.168.1.11)

The fundamental issue stems from SSH's security model. Each server generates unique host keys during installation. When clients connect via the VIP:

  • First connection to server1: stores server1's key for myservice.mycompany.uk
  • Failover to server2: client detects different key for same hostname → security error

The proposed solution of maintaining both keys in known_hosts is technically sound. Here's how to implement it across different platforms:

For OpenSSH Clients (Linux/macOS)

# Add both server keys to known_hosts
ssh-keyscan -t rsa server1 >> ~/.ssh/known_hosts
ssh-keyscan -t rsa server2 >> ~/.ssh/known_hosts

# Manually edit to replace IPs with VIP hostname
sed -i 's/server1/myservice.mycompany.uk/g' ~/.ssh/known_hosts
sed -i 's/server2/myservice.mycompany.uk/g' ~/.ssh/known_hosts

For PuTTY/PSFTP (Windows)

Windows registry stores keys differently. You'll need to:

  1. Export registry entries for both servers from HKEY_CURRENT_USER\Software\SimonTatham\PuTTY\SshHostKeys
  2. Modify the exported .reg files to use the VIP hostname
  3. Merge both modified files back into registry

Shared Host Key Solution

For controlled environments, you can configure identical host keys:

# On primary server:
sudo cp /etc/ssh/ssh_host_* /shared/storage/

# On secondary server:
sudo cp /shared/storage/ssh_host_* /etc/ssh/
sudo systemctl restart sshd

DNS-based Solution

Use round-robin DNS with short TTL instead of VIP:

myservice.mycompany.uk. 300 IN A 192.168.1.10
myservice.mycompany.uk. 300 IN A 192.168.1.11

While sharing host keys works, it has security implications:

  • Compromise of one server affects trust in both
  • Key rotation becomes more complex
  • Consider using certificates instead of host keys

Here's a bash script to maintain known_hosts for VIP failover:

#!/bin/bash
VIP="myservice.mycompany.uk"
SERVERS=("server1" "server2")

# Backup existing file
cp ~/.ssh/known_hosts ~/.ssh/known_hosts.bak

# Process each server
for server in "${SERVERS[@]}"; do
  KEY=$(ssh-keyscan -t rsa $server 2>/dev/null | awk '{print $2,$3}')
  if grep -q "$VIP" ~/.ssh/known_hosts; then
    sed -i "/$VIP/d" ~/.ssh/known_hosts
  fi
  echo "$VIP $KEY" >> ~/.ssh/known_hosts
done