How to Log All Linux Commands to a Remote Syslog Server (Bash/SSH Focus)


2 views

When implementing comprehensive command logging in Linux, we need to capture:

  • Interactive shell commands (bash being primary)
  • SSH session activities
  • Direct console inputs
  • Privileged command execution

For bash users, we can leverage PROMPT_COMMAND to send commands to syslog in real-time:

# Add to /etc/bash.bashrc or user's .bashrc
export PROMPT_COMMAND='RETRN_VAL=$?;logger -p local6.debug "$(whoami) [$$]: $(history 1 | sed "s/^[ ]*[0-9]\+[ ]*//") [$RETRN_VAL]"'

Configure syslog-ng to handle these messages:

# /etc/syslog-ng/conf.d/commands.conf
filter f_commands { facility(local6) and level(debug); };
destination remote_commands { syslog("logserver.example.com" port(514)); };
log { source(s_src); filter(f_commands); destination(remote_commands); };

For comprehensive SSH logging, configure sshd to log input/output:

# /etc/ssh/sshd_config
LogLevel VERBOSE
SyslogFacility AUTH
PrintMotd no
PrintLastLog no

Combine with pam_exec for session tracking:

# /etc/pam.d/sshd
session required pam_exec.so /usr/local/bin/log_session.sh

For capturing console commands, use auditd:

# /etc/audit/rules.d/commands.rules
-a exit,always -F arch=b64 -S execve -k command_logging
-a exit,always -F arch=b32 -S execve -k command_logging

Forward audit logs to syslog:

# /etc/audit/plugins.d/syslog.conf
active = yes
direction = out
path = builtin_syslog
format = string
args = LOG_LOCAL6 LOG_NOTICE

To catch potential circumvention attempts:

  • Monitor /tmp for custom shell installations
  • Log setuid command executions separately
  • Track process ancestry for suspicious patterns
# Example script to detect custom shells
#!/bin/bash
find / -type f -perm -u=x -exec file {} + | grep -i "shell script"

On your logserver, ensure proper parsing of command logs:

# Sample syslog-ng parser for command logs
parser p_command {
    csv-parser(
        columns("DATE", "HOST", "PROG", "PID", "USER", "SESSION", "COMMAND", "RETURN")
        delimiters(" ")
        quote-pairs('""[]')
        flags(strip-whitespace)
    );
};

When implementing comprehensive command auditing on Linux systems, we need to capture all interactive shell commands regardless of whether users connect via SSH or local console. The solution must work with bash (the default shell for most distributions) while being resilient against basic evasion attempts.

Before implementing command logging, ensure your syslog-ng is properly configured to forward logs to your central server. A basic working configuration should include:

destination logserver {
    syslog("logserver.example.com" port(514));
};
log {
    source(src);
    destination(logserver);
};

For bash shells, we can leverage the PROMPT_COMMAND environment variable to log each command immediately after execution:

# Add to /etc/profile or /etc/bash.bashrc
export PROMPT_COMMAND='RETRN_VAL=$?;logger -p local6.notice "$(whoami) [$$]: $(history 1 | sed "s/^[ ]*[0-9]\+[ ]*//" ) [$RETRN_VAL]"'

This will log commands with:

  • Username
  • Process ID
  • Command text
  • Return value

For more robust auditing that works across all shells, implement auditd rules:

# /etc/audit/rules.d/command-logging.rules
-a exit,always -F arch=b64 -S execve -k cmd-audit
-a exit,always -F arch=b32 -S execve -k cmd-audit

Then configure auditd to forward to syslog:

# /etc/audit/plugins.d/syslog.conf
active = yes
direction = out
path = builtin_syslog
format = string
args = LOG_LOCAL6 LOG_NOTICE

For SSH sessions specifically, configure the sshd server to log command execution:

# /etc/ssh/sshd_config
LogLevel VERBOSE
PrintLastLog yes

For local terminal sessions, configure script recording:

# /etc/profile.d/command-logging.sh
if [ "$PS1" ]; then
    exec script -q -f /var/log/terminal-sessions/$(date +%Y%m%d-%H%M%S)-$(whoami)-$$.log
fi

Ensure your logging solution includes proper log rotation to prevent disk exhaustion:

# /etc/logrotate.d/command_logs
/var/log/terminal-sessions/*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 640 root adm
}

To make the logging more tamper-resistant:

  • Set immutable flags on critical config files: chattr +i /etc/profile.d/command-logging.sh
  • Configure centralized logging with immediate forwarding
  • Restrict access to logging directories

After implementation, verify logging works by:

# Execute test commands
id
ls -la

# Check local logs
grep cmd-audit /var/log/audit/audit.log

# Verify remote logging
ssh logserver "grep $(whoami) /var/log/remote/commands.log"