Debugging SSH Key Authentication: “Not a RSA1 key file” and Unknown Key Type Errors in BackupPC


2 views

When your BackupPC server suddenly stops authenticating with SSH keys that have worked for years, the debug output reveals two critical error patterns:

debug3: Not a RSA1 key file /var/lib/BackupPC/.ssh/id_rsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'

The repeated "missing whitespace" debug messages strongly indicate either:

  • File corruption during storage/transfer
  • Inadvertent modification by text editors
  • Filesystem errors affecting the key

Verify the key integrity with:

# Check file encoding
file /var/lib/BackupPC/.ssh/id_rsa

# Validate RSA key structure
openssl rsa -in /var/lib/BackupPC/.ssh/id_rsa -check -noout

While permissions appear correct (600 for key files), the debug output shows the SSH client is attempting multiple key formats:

debug1: identity file /var/lib/BackupPC/.ssh/identity type -1
debug1: identity file /var/lib/BackupPC/.ssh/id_rsa type 1
debug1: identity file /var/lib/BackupPC/.ssh/id_dsa type -1

Create a minimal SSH config to force RSA key usage:

# /var/lib/BackupPC/.ssh/config
Host *
    IdentitiesOnly yes
    IdentityFile ~/.ssh/id_rsa
    PreferredAuthentications publickey

When physical access isn't possible, generate a new key pair on the working root account and deploy it securely:

# On backup server as root:
ssh-keygen -t rsa -b 4096 -f /tmp/backuppc_temp
cat /tmp/backuppc_temp.pub | ssh remote_host "sudo tee -a ~backuppc/.ssh/authorized_keys"

# Set proper ownership remotely:
ssh remote_host "sudo chown backuppc:backuppc ~backuppc/.ssh/authorized_keys"

For maximum reliability when transferring keys between systems:

# On source machine:
openssl rsa -in id_rsa -outform DER | base64 > id_rsa.der64

# On destination:
base64 -d id_rsa.der64 | openssl rsa -inform DER -out id_rsa
chmod 600 id_rsa

As temporary solution while troubleshooting:

# On backup server:
eval $(ssh-agent)
ssh-add /var/lib/BackupPC/.ssh/id_rsa
ssh -A remote_host

When attempting SSH authentication as the backuppc user, we encounter two distinct errors in debug output:

debug3: Not a RSA1 key file /var/lib/BackupPC/.ssh/id_rsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'

The first error suggests OpenSSH is expecting the older RSA1 format (SSH protocol 1), while the second indicates parsing failure of the key file header.

Modern OpenSSH typically uses PEM format for private keys with headers like:

-----BEGIN RSA PRIVATE KEY-----
[base64-encoded key data]
-----END RSA PRIVATE KEY-----

Let's verify the actual key format:

# Check first 3 lines of the private key
head -3 /var/lib/BackupPC/.ssh/id_rsa

While permissions appear correct (600 for private key), there's a subtle ownership issue:

drwxr-x--- 9 backuppc root      125 May 12 19:23 ..

The parent directory is owned by root but should belong to backuppc for proper SSH operation.

Potential issues in SSH client configuration:

# Check for protocol version forcing
grep -i "protocol" /var/lib/BackupPC/.ssh/config

Older systems might have conflicting settings in /etc/ssh/ssh_config:

# Example of problematic configuration
Host *
    Protocol 2,1
    IdentityFile ~/.ssh/identity
    IdentityFile ~/.ssh/id_rsa
    IdentityFile ~/.ssh/id_dsa

First, correct the parent directory ownership:

sudo chown backuppc:backuppc /var/lib/BackupPC/

Then regenerate the public key from existing private key:

ssh-keygen -y -f /var/lib/BackupPC/.ssh/id_rsa > /var/lib/BackupPC/.ssh/id_rsa.pub

For systems requiring RSA1 format (rare):

ssh-keygen -t rsa1 -f /var/lib/BackupPC/.ssh/id_rsa1 -C "backuppc RSA1 key"

Use verbose mode with specific configuration:

ssh -vvv -F /var/lib/BackupPC/.ssh/config -i /var/lib/BackupPC/.ssh/id_rsa backuppc@remote-host

If key format conversion is needed (PEM to OpenSSH):

# Convert RSA private key to PEM format
openssl rsa -in id_rsa -outform pem > id_rsa.pem

# Convert between SSH key formats
ssh-keygen -p -m PEM -f /var/lib/BackupPC/.ssh/id_rsa

On remote servers, verify these settings in /etc/ssh/sshd_config:

PubkeyAuthentication yes
RSAAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

Remember to restart sshd after changes:

sudo systemctl restart sshd