Troubleshooting GitLab SSH Key Authentication Issues: Permission Denied and Connection Closed Errors


11 views

When working with GitLab, SSH key authentication issues can be particularly frustrating because they prevent basic Git operations like clone, pull, and push. The symptoms you're seeing - 403 errors with HTTP and "permission denied" with SSH - typically indicate authentication problems.

First, let's verify some critical components:

# Check if SSH daemon is running
sudo systemctl status sshd

# Verify GitLab's SSH configuration
cat /etc/gitlab/gitlab.rb | grep ssh

The SSH config file (~/.ssh/config) you've created looks correct, but there might be permission issues:

# Verify file permissions
ls -la ~/.ssh/

# Correct permissions if needed
chmod 600 ~/.ssh/config
chmod 600 ~/.ssh/id_rsa

The /var/log/secure entries showing "Connection closed" suggest the SSH handshake is failing. Let's enable more verbose logging:

# Edit SSH daemon config
sudo vim /etc/ssh/sshd_config

# Add or modify these lines:
LogLevel VERBOSE
PubkeyAuthentication yes

Before trying Git operations, test the SSH connection directly:

ssh -Tv git@your-gitlab-server.com

This verbose output (-v) will show exactly where the authentication fails.

For GitLab installations, there are some special considerations:

# Reconfigure GitLab after changes
sudo gitlab-ctl reconfigure

# Check GitLab's authorized_keys file
sudo cat /var/opt/gitlab/.ssh/authorized_keys

If you're using Windows with Git Bash, try:

# Start SSH agent
eval $(ssh-agent -s)
ssh-add ~/.ssh/id_rsa

If SSH continues to fail, consider temporary alternatives:

# Use HTTPS with credential caching
git config --global credential.helper cache
git clone https://your-gitlab-server.com/your-repo.git

After making changes, verify everything works:

# Test Git operations
git clone git@your-gitlab-server.com:your-repo.git
cd your-repo
touch test.txt
git add test.txt
git commit -m "Test commit"
git push origin master

The problem manifests when trying to perform Git operations (clone/pull/push) against a GitLab instance where:

  • Web UI functions normally
  • SSH connections get terminated abruptly (Connection closed by [IP])
  • HTTP attempts result in 403 errors
  • Public keys appear properly registered in GitLab

First let's verify the SSH client configuration. The provided ~/.ssh/config appears correct, but we should add debug flags:

Host gitlab.example.com
    User git
    Hostname gitlab.example.com
    PreferredAuthentications publickey
    IdentityFile ~/.ssh/id_rsa
    LogLevel DEBUG3

The /var/log/secure logs show connection closures without authentication attempts. This suggests either:

  1. SSH daemon configuration issues
  2. Filesystem permission problems
  3. GitLab's authorized_keys handling

Check the GitLab Shell logs:

sudo tail -f /var/log/gitlab/gitlab-shell/gitlab-shell.log

Run these diagnostic commands on the GitLab server:

# Verify git user's home directory permissions
sudo ls -ld /var/opt/gitlab/
sudo ls -la /var/opt/gitlab/.ssh/

# Check GitLab's authorized_keys file
sudo cat /var/opt/gitlab/.ssh/authorized_keys

# Validate SELinux context if applicable
sudo ls -Z /var/opt/gitlab/.ssh/authorized_keys

Common solutions include:

# Regenerate authorized_keys
sudo gitlab-rake gitlab:shell:setup

# Reconfigure GitLab
sudo gitlab-ctl reconfigure

# Verify SSH daemon configuration
sudo grep -i "AcceptEnv" /etc/ssh/sshd_config

Test SSH connectivity manually:

ssh -Tv git@gitlab.example.com
# If connection succeeds but Git operations fail:
GIT_SSH_COMMAND="ssh -v" git clone git@gitlab.example.com:user/repo.git

For Windows clients using Pageant:

  1. Ensure Pageant is running with your private key loaded
  2. Verify GIT_SSH environment variable points to plink.exe
  3. Check for line ending issues in keys:
# In git-bash:
dos2unix ~/.ssh/id_rsa

If SELinux is enforcing:

# Check for denials
sudo ausearch -m avc -ts recent
# Temporary disable to test
sudo setenforce 0
# If problem resolves:
sudo semanage fcontext -a -t ssh_home_t "/var/opt/gitlab/.ssh/authorized_keys"
sudo restorecon -v /var/opt/gitlab/.ssh/authorized_keys