Troubleshooting SSH Connection Refused After Google Compute Engine Disk Resize and Restart


2 views


The "Connection Refused" error on port 22 typically indicates one of these scenarios:

  1. The SSH daemon isn't running on the VM
  2. Firewall rules are blocking the connection
  3. The network interface configuration is incorrect
  4. Disk resize caused filesystem issues affecting SSH configuration

# Check if SSH service is running through serial console
sudo systemctl status sshd

# Check listening ports
sudo netstat -tuln | grep 22

# Verify GCP firewall rules
gcloud compute firewall-rules list --filter="name~allow-ssh"

Based on your serial console output showing boot errors, try these solutions:


# 1. Resize filesystem through serial console (if ext4)
sudo resize2fs /dev/sda1

# 2. Repair potential filesystem corruption
sudo fsck -y /dev/sda1

# 3. Reinstall SSH server if configuration corrupted
sudo apt-get purge openssh-server
sudo apt-get install openssh-server

If basic fixes don't work, attach the disk to another VM:


# Create a new temporary VM
gcloud compute instances create recovery-vm --zone=your-zone

# Attach the problematic disk
gcloud compute instances attach-disk recovery-vm \
  --disk=your-problem-disk \
  --device-name=recovery-disk

# Mount and examine filesystem
sudo mkdir /mnt/recovery
sudo mount /dev/disk/by-id/google-recovery-disk-part1 /mnt/recovery
sudo chroot /mnt/recovery

# Inside chroot environment:
nano /etc/ssh/sshd_config  # Verify configuration
systemctl enable ssh       # Ensure service starts on boot
exit

Add these safeguards to your deployment scripts:


#!/bin/bash
# Post-resize verification script
check_ssh_active() {
  if ! systemctl is-active --quiet ssh; then
    logger "SSH service not active after resize"
    systemctl restart ssh
    exit 1
  fi
}

check_disk_mount() {
  if ! findmnt / >/dev/null; then
    logger "Root filesystem not mounted properly"
    mount -a
    resize2fs /dev/sda1
  fi
}

check_ssh_active
check_disk_mount

When attempting to SSH into my GCE instance after resizing the boot disk and restarting, I encountered a "Connection refused" error both through Google Cloud Shell and PuTTY. The serial console output (full log here) showed the system booted normally but SSH service wasn't accessible.

Before diving deep, let's verify some basics:

# Check if SSH daemon is running (from serial console)
sudo systemctl status sshd

# Alternative check for older systems
service ssh status

# Verify listening ports
netstat -tuln | grep 22
ss -tuln | grep 22

From experience, these are the most likely culprits after disk operations:

  • Corrupted /etc/ssh/sshd_config during resize
  • Firewall rules being reset
  • Missing iptables rules
  • Disk mounting issues affecting service startup

Check GCP firewall rules and instance-level settings:

# Check GCP firewall (from Cloud Shell)
gcloud compute firewall-rules list --filter="name~allow-ssh"

# Verify instance metadata
gcloud compute instances describe [INSTANCE_NAME] --zone=[ZONE] | grep -A5 metadata

If basic checks don't help, try these approaches:

Method 1: Temporary Startup Script

# Add this metadata key-value pair to your instance
startup-script: |
  #!/bin/bash
  apt-get update
  apt-get install --reinstall openssh-server
  systemctl restart ssh

Method 2: Manual Config Repair via Serial Console

# Mount the disk if needed
mount /dev/sda1 /mnt

# Check SSH config
nano /mnt/etc/ssh/sshd_config

# Typical working config:
Port 22
Protocol 2
PermitRootLogin prohibit-password
PasswordAuthentication no

For future operations:

  1. Always take snapshots before disk modifications
  2. Create a backup SSH config file
  3. Consider using OS Login for more reliable access

If SSH remains inaccessible:

  • Use the serial console for emergency access
  • Create a new instance and attach the disk
  • Consider using gcloud beta scp for file transfers