When implementing a bastion host architecture in AWS VPC environments, teams often struggle with secure credential distribution. The standard workflow requires:

# Standard SSH connection flow with agent forwarding
ssh -A -i production_key.pem ec2-user@bastion.example.com
ssh ec2-user@internal-app-server

Effective SSH key management must address three critical aspects:

• Discovery: Clear mapping between environments and corresponding keys
• Storage: FIPS 140-2 compliant encryption for private keys
• Authorization: Granular access controls with audit trails

Vault-Based Solution (HashiCorp Vault Example)

Configure Vault's SSH secrets engine for dynamic credential generation:

# Enable SSH secrets engine
vault secrets enable -path=ssh-client-signer ssh

# Create CA role
vault write ssh-client-signer/roles/team-role \
  allow_user_certificates=true \
  allowed_users="ec2-user" \
  default_extensions=permit-pty \
  key_type=ca

AWS Systems Manager Parameter Store

Store encrypted keys with IAM access policies:

# Store encrypted PEM file
aws ssm put-parameter \
  --name "/prod/ssh_keys/bastion_key" \
  --value "$(cat bastion_prod.pem)" \
  --type SecureString \
  --key-id alias/aws/ssm

Git-Crypt for Team Repositories

Combine version control with transparent encryption:

# Initialize git-crypt
git-crypt init
echo "*.pem filter=git-crypt diff=git-crypt" >> .gitattributes

# Add team members
git-crypt add-gpg-user team-member@example.com

For teams using OpenSSH certificates:

# Generate user certificate
ssh-keygen -s ca_key -I user_identity -n ec2-user \
  -V +1d -z 1234 user_key.pub

• Rotate keys quarterly using AWS KMS key rotation
• Implement mandatory key passphrases with ssh-agent
• Enforce MFA for all credential access operations
• Maintain detailed CloudTrail logs for all key access

Sample AWS CLI command to audit key access:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=GetParameter \
  --query "Events[*].CloudTrailEvent" \
  --output text | jq .

---

When operating in AWS environments with bastion host architectures, securely distributing .pem files among team members presents several technical challenges:

• Key version control becomes critical as team members join/leave projects
• Unauthorized key copying creates security blind spots
• Multiple environments (dev/stage/prod) require different access levels

Here are three proven approaches with their technical implementations:

1. SSH Certificate Authority (Enterprise Solution)

# On CA server:
ssh-keygen -t rsa -f ca_key -C "Company SSH CA"

# For each user:
ssh-keygen -s ca_key -I user@company -n dev-team,staging user_key.pub

Advantages:

• Single certificate signs all access keys
• Automatic expiration built into certificates
• Centralized revocation capability

2. Encrypted Git Repository (DevOps Approach)

# Using git-crypt for encryption:
git-crypt init
echo "*.pem filter=git-crypt diff=git-crypt" >> .gitattributes
git-crypt add-gpg-user user@company.com

Implementation notes:

• Requires GPG key distribution to team members
• Allows granular file-level encryption
• Integrates with existing CI/CD pipelines

3. Vault-Based Key Management (Cloud Native)

# AWS Secrets Manager CLI example:
aws secretsmanager create-secret \
  --name production/ssh_keys \
  --secret-string file://prod_key.pem

Access control via IAM policies:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "secretsmanager:ResourceTag/project": "ecommerce"
        }
      }
    }
  ]
}

Regardless of the chosen method, implement these security measures:

• Rotate keys quarterly using AWS CLI: aws ec2 create-key-pair
• Set strict file permissions: chmod 600 ~/.ssh/company_keys/*.pem
• Use SSH config to prevent key leaks:

Host bastion-prod
  HostName prod-bastion.example.com
  User ec2-user
  IdentityFile ~/.ssh/company_keys/prod_key.pem
  ForwardAgent no

Implement these monitoring practices:

# CloudTrail query for SSH key access
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=GetSecretValue \
  --start-time "2023-01-01T00:00:00Z"

Key metrics to track:

• Failed authentication attempts
• Key rotation compliance
• Geolocation of access patterns