Automated SSH Host Key Management for Dynamic EC2 Instances: Preventing “REMOTE HOST IDENTIFICATION HAS CHANGED” Warnings


2 views

When working with dynamically provisioned EC2 instances, the SSH key verification system becomes a double-edged sword. Every time you terminate and recreate an instance with the same hostname or IP, you're greeted with that dreaded warning:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

The SSH client maintains a ~/.ssh/known_hosts file that stores host keys for verification. When you recreate an EC2 instance:

  1. A new SSH host key pair is generated
  2. The IP/hostname remains the same but points to new hardware
  3. SSH detects a mismatch between stored and presented keys

While ssh-keygen -R removes entries by hostname, it doesn't handle these cases effectively:

  • Multiple entries for the same IP (IPv4 and IPv6)
  • DNS names that resolve to the same IP
  • Hashed hostname entries (default in newer OpenSSH versions)

For scripting environments, here are robust approaches:

Method 1: Purge All Related Entries

#!/bin/bash
HOST="ec2-54-123-45-67.compute-1.amazonaws.com"
IP="54.123.45.67"

# Remove by hostname and IP
ssh-keygen -R "$HOST" &>/dev/null
ssh-keygen -R "$IP" &>/dev/null

# Nuclear option for hashed entries
sed -i "/$HOST/d" ~/.ssh/known_hosts
sed -i "/$IP/d" ~/.ssh/known_hosts

Method 2: Temporary Disable Strict Checking

For one-off connections in scripts:

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null user@host

Method 3: AWS-Specific Solution

Using AWS CLI to get instance metadata:

INSTANCE_ID="i-0123456789abcdef"
HOST=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID \
       --query 'Reservations[0].Instances[0].PublicDnsName' --output text)
IP=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID \
     --query 'Reservations[0].Instances[0].PublicIpAddress' --output text)

# Clean up existing entries
ssh-keygen -f ~/.ssh/known_hosts -R "$HOST"
ssh-keygen -f ~/.ssh/known_hosts -R "$IP"
  • Use EC2 Instance Connect for temporary access
  • Consider maintaining a dedicated jump host with persistent keys
  • Implement SSH certificate authority for dynamic environments
  • Store known_hosts in memory for containerized environments:
# Dockerfile example
RUN echo "StrictHostKeyChecking no" >> /etc/ssh/ssh_config

For predictable environments, bake host keys into AMIs:

# In your Packer configuration
"provisioners": [
  {
    "type": "shell",
    "inline": [
      "ssh-keygen -A",
      "sudo cp /etc/ssh/ssh_host_* /tmp/"
    ]
  }
]

When managing dynamic EC2 instances that get frequently rebuilt, SSH host key conflicts become inevitable. The standard warning about changed host identification - while important for security - creates operational friction in automated environments.

The error persists because modern OpenSSH stores two separate entries for each host:

1. Hostname-based entry (e.g., ec2-203-0-113-25.compute-1.amazonaws.com)
2. IP-based entry (e.g., 203.0.113.25)

This one-liner removes all traces of a host from known_hosts:

ssh-keygen -R "hostname" && ssh-keygen -R "ip_address"

For AWS environments, integrate this into your instance creation script:

#!/bin/bash
INSTANCE_IP=$(aws ec2 run-instances ... | jq -r '.Instances[0].PublicIpAddress')
INSTANCE_DNS=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID | jq -r '.Reservations[0].Instances[0].PublicDnsName')

# Clean previous entries
ssh-keygen -R "$INSTANCE_DNS" &>/dev/null
ssh-keygen -R "$INSTANCE_IP" &>/dev/null

# Force new key acceptance
ssh -o StrictHostKeyChecking=accept-new ubuntu@$INSTANCE_IP

Create more robust handling via ~/.ssh/config:

Host *.amazonaws.com
    StrictHostKeyChecking accept-new
    UserKnownHostsFile ~/.ssh/aws_hosts
    LogLevel ERROR

While automating, maintain security by:

  • Using separate known_hosts files for dynamic environments
  • Setting appropriate expiration with ssh-keyscan -T 30
  • Implementing host certificate authentication where possible