How to Format AWS Secrets Manager SSH Private Keys into Valid PEM Format Using sed/awk


2 views

When storing SSH private keys in AWS Secrets Manager, you'll encounter an annoying formatting issue - the service automatically converts multi-line PEM keys into a single line. This breaks the key's validity since SSH clients expect proper PEM formatting:

# Current (invalid) format from Secrets Manager:
-----BEGIN RSA PRIVATE KEY----- MIIEpAIBAAKCAQEA...base64data... -----END RSA PRIVATE KEY-----

# Required (valid) PEM format:
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEA...base64data...
-----END RSA PRIVATE KEY-----

The Privacy Enhanced Mail (PEM) format requires specific headers/footers and line breaks. SSH clients like OpenSSH strictly validate this structure. AWS Secrets Manager's single-line storage breaks two critical requirements:

  • Missing newlines after BEGIN/END markers
  • Base64 data exceeding 64-character line length

Here are three reliable methods to reformat your key:

1. Using sed (Most Universal)

# Basic version:
sed 's/ /\n/g' input.key > output.key

# More robust version handling extra spaces:
sed -E 's/^-----BEGIN[^-]+----- //; s/ -----END[^-]+-----$//; s/ /\n/g' \
    | sed '1i-----BEGIN RSA PRIVATE KEY-----' \
    | sed '$a-----END RSA PRIVATE KEY-----'

2. Using awk (Better for Complex Cases)

awk '{
  gsub(/-----BEGIN[^-]+----- /,"");
  gsub(/ -----END[^-]+-----/,"");
  gsub(/ /,"\n");
  print "-----BEGIN RSA PRIVATE KEY-----\n" $0 "\n-----END RSA PRIVATE KEY-----"
}' input.key > output.key

3. Python Script (Most Maintainable)

#!/usr/bin/env python3
import re

def reformat_key(one_line_key):
    # Extract base64 content
    match = re.match(
        r'-----BEGIN[^-]+-----(.+)-----END[^-]+-----',
        one_line_key.strip()
    )
    if not match:
        raise ValueError("Invalid key format")
    
    content = match.group(1).strip()
    return f"-----BEGIN RSA PRIVATE KEY-----\n{content}\n-----END RSA PRIVATE KEY-----"

print(reformat_key(sys.stdin.read()))

For seamless integration with AWS Secrets Manager:

# Get and reformat key in one command
aws secretsmanager get-secret-value \
    --secret-id my-ssh-key \
    --query SecretString \
    --output text \
    | sed 's/ /\n/g' > ~/.ssh/id_rsa

# Set correct permissions
chmod 600 ~/.ssh/id_rsa
  • Handle different key types (ECDSA, ED25519)
  • Account for potential extra whitespace
  • Verify line lengths don't exceed 64 chars

When storing SSH private keys in AWS Secrets Manager, the service automatically flattens the content into a single line. This breaks the standard PEM format that SSH clients expect:

# Current format (invalid)
-----BEGIN RSA PRIVATE KEY----- MIIEpQIBAAKCAQEA... -----END RSA PRIVATE KEY-----

# Required format (valid)
-----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEA...
-----END RSA PRIVATE KEY-----

Here's how to fix this using sed to insert newlines:

# Basic sed solution
sed 's/ -----BEGIN RSA PRIVATE KEY----- /&\n/;s/ -----END RSA PRIVATE KEY----- /\n&/' single_line_key > fixed_key.pem

For a more robust version that handles different key types:

# Advanced sed with pattern matching
sed -E 's/(-----(BEGIN|END) (RSA|OPENSSH|EC) PRIVATE KEY-----)/\n\1\n/g' \
    | sed '/^$/d' > formatted_key.pem

Here's an awk alternative that's more readable:

awk '{
    gsub(" -----BEGIN RSA PRIVATE KEY----- ", "&\n");
    gsub(" -----END RSA PRIVATE KEY----- ", "\n&");
    print
}' input_key > output_key.pem

If you're working with AWS Lambda, here's a Python solution:

import re

def format_ssh_key(raw_key):
    return re.sub(
        r'(-----(BEGIN|END) .*? PRIVATE KEY-----)',
        r'\n\1\n',
        raw_key
    ).strip()

# Usage example:
formatted_key = format_ssh_key(secrets_manager_value)

After conversion, verify your key with:

ssh-keygen -y -f fixed_key.pem > public_key.pub

If this command succeeds without errors, your key is properly formatted.

For a complete AWS CLI solution that retrieves, formats, and saves the key:

aws secretsmanager get-secret-value --secret-id my-ssh-key \
    --query SecretString --output text \
    | sed 's/ -----BEGIN RSA PRIVATE KEY----- /&\n/' \
    | sed 's/ -----END RSA PRIVATE KEY----- /\n&/' \
    > ~/.ssh/id_rsa

chmod 600 ~/.ssh/id_rsa