When debugging non-interactive SSH sessions where environment variables aren't being sourced properly, we often encounter a frustrating scenario where commands work in interactive shells but fail during remote execution. The key culprit is usually how $BASH_ENV gets set (or not set) for non-interactive bash sessions.

Bash follows a specific order when determining environment configuration:

1. For interactive login shells:
   - /etc/profile
   - ~/.bash_profile, ~/.bash_login, ~/.profile (first found)
   
2. For non-interactive shells:
   - Checks $BASH_ENV (if set)
   - Otherwise skips most initialization files

Based on the server setup you described, these are the most likely places to investigate:

# 1. System-wide config (affects all users)
/etc/environment
/etc/profile.d/custom.sh

# 2. User-specific files
~/.ssh/environment
~/.bash_profile
~/.bashrc (though this usually doesn't set BASH_ENV itself)

# 3. PAM modules (advanced cases)
/etc/pam.d/sshd

To track down where $BASH_ENV is being set on your working server:

# Method 1: Search through common config files
grep -r "BASH_ENV" /etc/ ~/.ssh/ ~/

# Method 2: Debug bash startup
env -i bash --noprofile --norc -c 'echo $BASH_ENV'

# Method 3: Check PAM environment
grep -i environment /etc/pam.d/*

The most reliable approach is to explicitly set BASH_ENV in /etc/environment:

# Add this line to /etc/environment
BASH_ENV="/home/username/.bashrc"

Or alternatively in a custom profile script:

# Create /etc/profile.d/set_bash_env.sh
#!/bin/bash
export BASH_ENV="$HOME/.bashrc"

If you're using SSH, ensure these settings are in place:

# In /etc/ssh/sshd_config:
PermitUserEnvironment yes

# Then in ~/.ssh/environment:
BASH_ENV=~/.bashrc
PATH=/usr/local/bin:/usr/bin:/bin:/path/to/your/app/bin

Verify the fix works with:

ssh user@server 'env | grep BASH_ENV'
ssh user@server '/path/to/your/command'

Remember that changes to /etc/environment or PAM configs may require a reboot, while SSH config changes need systemctl restart sshd.

When troubleshooting environment variable issues across SSH connections, I recently encountered a fascinating case where two supposedly identical Linux servers behaved differently. The core symptom: commands relying on ~/.bashrc paths worked interactively but failed during SSH execution on one server.

After thorough examination, several critical points emerged:

# On working server:
$ echo $BASH_ENV
/home/user/.bashrc

# On non-working server:
$ echo $BASH_ENV
(empty string)

Both systems shared identical configurations in:

• /etc/profile
• /etc/environment
• ~/.bash_profile (containing: if [ -f ~/.bashrc ]; then . ~/.bashrc; fi)
• ~/.ssh/environment (with BASH_ENV=~/.bashrc)

For SSH commands (non-interactive login shells), Bash follows this initialization sequence:

1. /etc/profile
2. ~/.bash_profile
3. Files specified in $BASH_ENV (if set)
4. ~/.ssh/environment (when PermitUserEnvironment is enabled)

Through process tracing (strace), I discovered the working server had this configuration in an unexpected location:

# Found in /etc/bash.bashrc (Debian/Ubuntu) or /etc/bashrc (RHEL/CentOS)
if [ "$PS1" ]; then
    # Interactive shell configuration
else
    # For non-interactive shells
    export BASH_ENV=~/.bashrc
fi

To ensure consistent behavior across servers, implement this comprehensive solution:

# 1. Check if PermitUserEnvironment is enabled
sudo grep -q 'PermitUserEnvironment yes' /etc/ssh/sshd_config || \
    echo "PermitUserEnvironment yes" | sudo tee -a /etc/ssh/sshd_config

# 2. Ensure ~/.ssh/environment exists and has proper permissions
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "BASH_ENV=$HOME/.bashrc" > ~/.ssh/environment
chmod 600 ~/.ssh/environment

# 3. Add system-wide configuration (choose appropriate file)
# For Debian/Ubuntu:
echo -e "\n# Set BASH_ENV for non-interactive shells\n[ -z \"$PS1\" ] && export BASH_ENV=$HOME/.bashrc" | \
    sudo tee -a /etc/bash.bashrc

# For RHEL/CentOS:
echo -e "\n# Set BASH_ENV for non-interactive shells\n[ -z \"$PS1\" ] && export BASH_ENV=$HOME/.bashrc" | \
    sudo tee -a /etc/bashrc

# 4. Restart SSH service
sudo systemctl restart sshd

Confirm the solution works with these test cases:

# Test direct command execution
ssh user@server "echo \$BASH_ENV"

# Test path-dependent command
ssh user@server "/path/to/your/custom/command"

# Compare environments
ssh user@server "env | sort" > ssh_env.txt
ssh user@server "bash -i -c 'env | sort'" > interactive_env.txt
diff ssh_env.txt interactive_env.txt