Understanding PATH Variable Discrepancies Between SSH Interactive Shells and Remote Commands


1 views

When executing commands via SSH, you might observe that the $PATH environment variable differs between interactive shells and direct remote commands. Consider these scenarios:

# Interactive shell
$ ssh example.com
user@example.com:~$ echo $PATH
/usr/local/bin:/usr/bin:/bin

# Remote command
$ ssh example.com 'echo $PATH'
/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games

The discrepancy stems from how SSH and shells handle initialization files:

  • Interactive login shells read /etc/profile and ~/.bash_profile
  • Non-interactive shells (like remote commands) might skip these files
  • SSH itself sets a default PATH when executing remote commands

To diagnose your specific case, examine these files:

# Check system-wide PATH settings
$ cat /etc/environment
$ cat /etc/profile

# Verify SSH environment handling
$ grep -i path /etc/ssh/sshd_config

Several approaches can ensure consistent PATH behavior:

1. Force PATH in .bashrc

# ~/.bashrc
if [ -z "$SSH_CONNECTION" ]; then
    export PATH="/usr/local/bin:/usr/bin:/bin"
fi

2. Use SSH Environment Passing

# In /etc/ssh/sshd_config
PermitUserEnvironment yes

# Then in ~/.ssh/environment
PATH=/usr/local/bin:/usr/bin:/bin

3. Explicit PATH in Remote Commands

$ ssh example.com 'PATH=/usr/local/bin:/usr/bin:/bin; your_command'

The exact behavior depends on:

  • SSH server configuration (AcceptEnv, PermitUserEnvironment)
  • Client-side SSH options (SendEnv)
  • Shell initialization files read for non-interactive sessions

For bash, you can force initialization file reading:

$ ssh -t example.com '/bin/bash --login -c "echo \$PATH"'

Here's how to properly test and implement a solution:

# 1. Test current PATH behavior
$ ssh example.com 'env | grep ^PATH='

# 2. Modify bash initialization
$ echo 'export PATH="/custom/path:$PATH"' >> ~/.bash_profile

# 3. Force login shell for remote commands
$ ssh example.com 'bash -l -c "your_command"'

When executing remote commands via SSH, many developers encounter a puzzling behavior where the $PATH environment variable differs from what they see in an interactive shell. Let's examine this through concrete examples:


# Interactive shell behavior
$ ssh example.com
user@example.com:~$ echo $PATH
/usr/local/bin:/usr/bin:/bin

# Remote command behavior
$ ssh example.com 'echo $PATH'
/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games

The divergence occurs because SSH handles these two scenarios differently:

  1. Interactive login shells process:
    • /etc/profile
    • ~/.bash_profile, ~/.bash_login, or ~/.profile
  2. Non-interactive remote commands typically only process:
    • ~/.bashrc (if the shell is bash)
    • The default PATH compiled into SSH

Here are several approaches to ensure consistent PATH behavior:

Option 1: Force Interactive Shell Behavior


ssh -t example.com '/tmp/hello.sh'

Option 2: Explicitly Set PATH in Remote Command


ssh example.com 'PATH=/usr/local/bin:/usr/bin:/bin; /tmp/hello.sh'

Option 3: Configure SSH Server

Modify /etc/ssh/sshd_config:


PermitUserEnvironment yes

Then create ~/.ssh/environment:


PATH=/usr/local/bin:/usr/bin:/bin

To understand exactly what's happening, enable shell tracing:


ssh example.com 'PS4="+ "; set -x; /tmp/hello.sh'

For all users, modify /etc/environment:


PATH="/usr/local/bin:/usr/bin:/bin"

This approach works because /etc/environment is processed by PAM, not the shell, ensuring consistent behavior across all SSH sessions.

This inconsistency particularly affects:

  • CI/CD pipelines
  • Cron jobs executed via SSH
  • Remote deployment scripts

Always verify the environment when writing automation scripts that rely on specific PATH settings.