Troubleshooting GitLab SSH Key Authentication Issues: Permission Denied and Connection Closed Errors


2 views

When working with GitLab, SSH key authentication issues can be particularly frustrating because they prevent basic Git operations like clone, pull, and push. The symptoms you're seeing - 403 errors with HTTP and "permission denied" with SSH - typically indicate authentication problems.

First, let's verify some critical components:

# Check if SSH daemon is running
sudo systemctl status sshd

# Verify GitLab's SSH configuration
cat /etc/gitlab/gitlab.rb | grep ssh

The SSH config file (~/.ssh/config) you've created looks correct, but there might be permission issues:

# Verify file permissions
ls -la ~/.ssh/

# Correct permissions if needed
chmod 600 ~/.ssh/config
chmod 600 ~/.ssh/id_rsa

The /var/log/secure entries showing "Connection closed" suggest the SSH handshake is failing. Let's enable more verbose logging:

# Edit SSH daemon config
sudo vim /etc/ssh/sshd_config

# Add or modify these lines:
LogLevel VERBOSE
PubkeyAuthentication yes

Before trying Git operations, test the SSH connection directly:

ssh -Tv git@your-gitlab-server.com

This verbose output (-v) will show exactly where the authentication fails.

For GitLab installations, there are some special considerations:

# Reconfigure GitLab after changes
sudo gitlab-ctl reconfigure

# Check GitLab's authorized_keys file
sudo cat /var/opt/gitlab/.ssh/authorized_keys

If you're using Windows with Git Bash, try:

# Start SSH agent
eval $(ssh-agent -s)
ssh-add ~/.ssh/id_rsa

If SSH continues to fail, consider temporary alternatives:

# Use HTTPS with credential caching
git config --global credential.helper cache
git clone https://your-gitlab-server.com/your-repo.git

After making changes, verify everything works:

# Test Git operations
git clone git@your-gitlab-server.com:your-repo.git
cd your-repo
touch test.txt
git add test.txt
git commit -m "Test commit"
git push origin master

The problem manifests when trying to perform Git operations (clone/pull/push) against a GitLab instance where:

  • Web UI functions normally
  • SSH connections get terminated abruptly (Connection closed by [IP])
  • HTTP attempts result in 403 errors
  • Public keys appear properly registered in GitLab

First let's verify the SSH client configuration. The provided ~/.ssh/config appears correct, but we should add debug flags:

Host gitlab.example.com
    User git
    Hostname gitlab.example.com
    PreferredAuthentications publickey
    IdentityFile ~/.ssh/id_rsa
    LogLevel DEBUG3

The /var/log/secure logs show connection closures without authentication attempts. This suggests either:

  1. SSH daemon configuration issues
  2. Filesystem permission problems
  3. GitLab's authorized_keys handling

Check the GitLab Shell logs:

sudo tail -f /var/log/gitlab/gitlab-shell/gitlab-shell.log

Run these diagnostic commands on the GitLab server:

# Verify git user's home directory permissions
sudo ls -ld /var/opt/gitlab/
sudo ls -la /var/opt/gitlab/.ssh/

# Check GitLab's authorized_keys file
sudo cat /var/opt/gitlab/.ssh/authorized_keys

# Validate SELinux context if applicable
sudo ls -Z /var/opt/gitlab/.ssh/authorized_keys

Common solutions include:

# Regenerate authorized_keys
sudo gitlab-rake gitlab:shell:setup

# Reconfigure GitLab
sudo gitlab-ctl reconfigure

# Verify SSH daemon configuration
sudo grep -i "AcceptEnv" /etc/ssh/sshd_config

Test SSH connectivity manually:

ssh -Tv git@gitlab.example.com
# If connection succeeds but Git operations fail:
GIT_SSH_COMMAND="ssh -v" git clone git@gitlab.example.com:user/repo.git

For Windows clients using Pageant:

  1. Ensure Pageant is running with your private key loaded
  2. Verify GIT_SSH environment variable points to plink.exe
  3. Check for line ending issues in keys:
# In git-bash:
dos2unix ~/.ssh/id_rsa

If SELinux is enforcing:

# Check for denials
sudo ausearch -m avc -ts recent
# Temporary disable to test
sudo setenforce 0
# If problem resolves:
sudo semanage fcontext -a -t ssh_home_t "/var/opt/gitlab/.ssh/authorized_keys"
sudo restorecon -v /var/opt/gitlab/.ssh/authorized_keys