Debugging SSH Push Failures in GitLab: Key Access Rights and Repository Permission Issues


2 views

When setting up a fresh GitLab instance (version 6.5.1), many administrators encounter the frustrating SSH push failure:

$ git push -u origin master
fatal: Could not read from remote repository.
Please make sure you have the correct access
rights and the repository exists.

Before diving deeper, let's acknowledge what's already been checked:

  • SSH key authentication is working (confirmed via debug logs)
  • The repository physically exists with correct permissions
  • GitLab-shell health check passes all tests
  • Service restarts don't resolve the issue
  • HTTP pushes work fine (LDAP authentication succeeds)

Here are some often-overlooked access right issues specific to GitLab 6.5.1:

1. GitLab-shell PATH Environment

The Ruby environment for gitlab-shell might not have proper PATH configuration. Check:

sudo -u gitlab -H env | grep PATH

Compare this with what's in /home/gitlab/.bashrc. A missing PATH to git binaries often causes silent failures.

2. Authorized Keys Command Mismatch

In config/gitlab.yml, verify:

gitlab_shell:
  path: /home/gitlab/gitlab-shell/
  authorized_keys: /home/gitlab/.ssh/authorized_keys
  repos_path: /home/gitlab/repositories/

The authorized_keys path must exactly match what's in /etc/ssh/sshd_config:

AuthorizedKeysFile /home/gitlab/.ssh/authorized_keys

3. Repository Namespace Permissions

Even if the repository exists, check:

ls -la /home/gitlab/repositories/group/subgroup/

The git user must have execute permissions on every directory in the path chain.

1. Enable GitLab-shell Verbose Logging

Edit /home/gitlab/gitlab-shell/config.yml:

log_level: DEBUG
log_file: /home/gitlab/gitlab-shell/gitlab-shell.log

Then monitor the log during a push attempt:

tail -f /home/gitlab/gitlab-shell/gitlab-shell.log

2. Test the Git Protocol Directly

Bypass SSH to isolate the issue:

sudo -u gitlab -H /home/gitlab/gitlab-shell/bin/gitlab-projects \
  add-project --name group/repo.git --path /home/gitlab/repositories/group/repo.git

3. SELinux Context Issues

On RHEL/CentOS systems:

ls -Z /home/gitlab/repositories/
restorecon -Rv /home/gitlab/repositories/

Here's a complete corrective script that addresses multiple potential issues:

#!/bin/bash
# Fix GitLab 6.5.1 SSH push permissions
chown -R gitlab:gitlab /home/gitlab/repositories
find /home/gitlab/repositories -type d -exec chmod 2770 {} \;
find /home/gitlab/repositories -type f -exec chmod 660 {} \;
chmod 750 /home/gitlab/gitlab-shell
chmod 644 /home/gitlab/.ssh/authorized_keys
echo 'export PATH=/usr/local/bin:/usr/bin:/bin:/usr/games' >> /home/gitlab/.bashrc
sudo -u gitlab -H bundle exec rake gitlab:check RAILS_ENV=production
service gitlab restart

As a last resort, regenerate the gitlab-shell configuration:

cd /home/gitlab/gitlab-shell
sudo -u gitlab -H ./bin/install

Then verify the API endpoint in config.yml matches your GitLab URL exactly.

When encountering the error fatal: Could not read from remote repository during GitLab SSH pushes, we're typically dealing with one of these scenarios:

$ git push -u origin master
fatal: Could not read from remote repository.

Please make sure you have the correct access
rights and the repository exists.

While the original post covers common solutions, let's dive deeper into the technical aspects:

SSH Keychain Verification

Even with a clean setup, verify the SSH agent is properly forwarding keys:

# Check loaded keys
ssh-add -l

# Test connection with verbose output
GIT_SSH_COMMAND="ssh -v" git push origin master

The gitlab-shell check might show OK, but these configuration files need verification:

# /home/git/gitlab-shell/config.yml
gitlab_url: "https://your.gitlab.domain/"
http_settings:
  self_signed_cert: false
repos_path: "/home/git/repositories"

Even with correct repository creation, these permission checks are crucial:

# Verify repository ownership
ls -la /home/git/repositories/group/project.git

# Should show git:git ownership and 2770 permissions
drwxrws--- 4 git git 4096 Jun 15 10:00 project.git

When standard checks don't reveal the issue, try these methods:

GitLab-Shell Execution Tracing

sudo -u git -H strace -f -o /tmp/gitlab-shell-trace.log \
  /home/git/gitlab-shell/bin/gitlab-shell key-123

Unicorn API Endpoint Testing

Directly test the API endpoint gitlab-shell uses:

curl -H "Content-Type: application/json" \
  http://localhost:8080/api/v4/internal/check

Understanding the exact command GitLab executes during push:

# Check the forced command in authorized_keys
command="/home/git/gitlab-shell/bin/gitlab-shell key-123",no-port-forwarding,\
no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3...== user@host

When all else fails, these workarounds might help:

# Temporary workaround using HTTP
git remote set-url origin https://gitlab.domain/group/project.git

# Or using explicit SSH path
git remote set-url origin ssh://git@gitlab.domain:22/home/git/repositories/group/project.git

Enable additional logging in these locations:

# /etc/gitlab/gitlab.rb (Omnibus) or equivalent
gitlab_shell['log_level'] = 'debug'
gitlab_workhorse['logging_verbose'] = true