The error message clearly indicates SSH authentication failure when trying to deploy via rsync in Gitlab CI/CD. The critical symptoms are:

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s)
Permission denied, please try again.
Permission denied (publickey,password).

The existing pipeline attempts to:

1. Generate a new SSH key during job execution
2. Add the server to known_hosts
3. Copy the newly generated public key to the server

This fails because:

• The $SERVER_USER account on the remote server doesn't allow password authentication
• Docker containers are ephemeral - new keys won't persist between jobs
• CI environments typically restrict interactive password prompts

Here's the correct way to implement SSH-based deployment:

1. Pre-configure SSH Keys

Create a dedicated deployment key pair beforehand:

ssh-keygen -t ed25519 -C "gitlab-ci-deploy-key" -f deploy_key

2. Configure Gitlab CI Variables

Add these as protected CI/CD variables:

• DEPLOY_SSH_PRIVATE_KEY (the private key contents)
• DEPLOY_SERVER_USER
• DEPLOY_SERVER_IP

3. Revised .gitlab-ci.yml

stages:
  - deploy

before_script:
  - npm install
  - npm run build
  # Set up SSH
  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh
  - echo "$DEPLOY_SSH_PRIVATE_KEY" > ~/.ssh/id_ed25519
  - chmod 600 ~/.ssh/id_ed25519
  - ssh-keyscan -H "$DEPLOY_SERVER_IP" >> ~/.ssh/known_hosts

job_deploy:
  stage: deploy 
  script:
    - rsync -avz --delete -e "ssh -o StrictHostKeyChecking=no" \
      $CI_PROJECT_DIR/dist/ \
      $DEPLOY_SERVER_USER@$DEPLOY_SERVER_IP:/var/www/example.com/
  only:
    - master

Using SSH Config File

For complex deployments, create an SSH config:

before_script:
  # ... previous setup ...
  - echo -e "Host deploy-target\n\
    HostName $DEPLOY_SERVER_IP\n\
    User $DEPLOY_SERVER_USER\n\
    IdentityFile ~/.ssh/id_ed25519\n\
    StrictHostKeyChecking no" > ~/.ssh/config

Verifying the Connection

Add a test step before deployment:

  - ssh deploy-target "echo 'SSH connection successful!'"

• Ensure the public key is in ~/.ssh/authorized_keys on the server
• Verify the remote user has write permissions to target directory
• Check SELinux/AppArmor permissions if files don't appear
• Use ssh -v for detailed connection debugging

---

When implementing continuous deployment using GitLab CI/CD pipelines, one common requirement is transferring build artifacts to production servers via SSH. The traditional approach of generating SSH keys during pipeline execution and attempting to copy them to the remote server often fails due to authentication issues.

Instead of generating keys during pipeline execution (which creates new keys each time), you should:

1. Generate SSH key pair locally
2. Add public key to remote server's authorized_keys
3. Store private key in GitLab CI/CD variables

# Generate key pair locally (not in CI)
ssh-keygen -t rsa -b 4096 -C "gitlab-ci@example.com" -f gitlab-deploy-key

In your GitLab project settings:

1. Add SSH_PRIVATE_KEY variable containing the private key content
2. Add SERVER_IP and SERVER_USER variables
3. Mark SSH_PRIVATE_KEY as "File" type for proper handling

Here's the improved pipeline configuration:

stages:
  - deploy

before_script:
  - apt-get update -y && apt-get install -y openssh-client rsync
  - mkdir -p ~/.ssh
  - echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
  - chmod 600 ~/.ssh/id_rsa
  - ssh-keyscan -H $SERVER_IP >> ~/.ssh/known_hosts

deploy_production:
  stage: deploy
  script:
    - npm install
    - npm run build
    - rsync -avz --delete -e "ssh -o StrictHostKeyChecking=no" \
      $CI_PROJECT_DIR/dist/ $SERVER_USER@$SERVER_IP:/var/www/example.com/
  only:
    - master

• Use project-level CI/CD variables (not group-level) for better isolation
• Restrict the deploy key to only necessary commands in ~/.ssh/authorized_keys
• Consider using SSH certificates for temporary access
• Rotate keys periodically (every 3-6 months)

If you encounter permission problems:

# Test SSH connection manually
ssh -T -i ~/.ssh/id_rsa $SERVER_USER@$SERVER_IP

# Verify remote directory permissions
ssh $SERVER_USER@$SERVER_IP "ls -ld /var/www/example.com"

# Check SSH daemon logs on remote server
tail -f /var/log/auth.log

For more complex deployments, create an SSH config file:

before_script:
  - echo "Host production" >> ~/.ssh/config
  - echo "  HostName $SERVER_IP" >> ~/.ssh/config
  - echo "  User $SERVER_USER" >> ~/.ssh/config
  - echo "  IdentityFile ~/.ssh/id_rsa" >> ~/.ssh/config
  - echo "  StrictHostKeyChecking no" >> ~/.ssh/config

# Then use in rsync:
rsync -avz --delete dist/ production:/var/www/example.com/