How to Log All Bash Script Output Including SSH Commands and Errors for Debugging


2 views

When debugging bash scripts, especially those involving remote commands via SSH, we often face the challenge of incomplete logging. The current approach using tee only captures explicitly echoed messages, missing crucial SSH output and potential error messages.

Bash handles three standard streams:

1. stdin (0) - Standard input
2. stdout (1) - Standard output
3. stderr (2) - Standard error

Your current solution only captures stdout from echo commands. SSH command outputs and errors go to different streams.

Here's an improved version that captures everything:

#!/bin/bash
{
    echo " $(date) : part 1 - start "
    ssh -f admin@server.com 'bash /www/htdocs/server.com/scripts/part1.sh logout exit'
    echo " $(date) : sleep 120"
    sleep 120
    echo " $(date) : part 2 - start"
    ssh admin@server.com 'bash /www/htdocs/server.com/scripts/part2.sh logout exit'
    echo " $(date) : part 3 - start"
    ssh admin@server.com 'bash /www/htdocs/server.com/scripts/part3.sh logout exit'
    echo " $(date) : END"
} 2>&1 | tee -a /home/scripts/cron/logs

2>&1 redirects stderr (2) to stdout (1), ensuring error messages are captured. The entire script block is wrapped in curly braces to apply the redirection to all commands.

For more complex scripts, consider using exec:

#!/bin/bash
log_file="/home/scripts/cron/logs"
exec > >(tee -a "${log_file}") 2>&1

echo " $(date) : part 1 - start "
ssh -f admin@server.com 'bash /www/htdocs/server.com/scripts/part1.sh logout exit'
# Rest of your script...

To verify SSH command success in your logs, use return code checks:

if ! ssh admin@server.com 'bash /path/to/script.sh'; then
    echo " $(date) : ERROR - SSH command failed"
    exit 1
fi

For better debugging, add millisecond precision timestamps:

#!/bin/bash
log_with_timestamp() {
    while IFS= read -r line; do
        printf '%(%Y-%m-%d %H:%M:%S)T %s\n' -1 "$line"
    done
}

{
    # Your commands here
} 2>&1 | log_with_timestamp | tee -a /home/scripts/cron/logs

For production systems, implement log rotation:

#!/bin/bash
LOG_DIR="/home/scripts/cron/logs"
MAX_LOG_SIZE=10485760 # 10MB

if [ -f "$LOG_DIR" ] && [ $(stat -c%s "$LOG_DIR") -gt $MAX_LOG_SIZE ]; then
    mv "$LOG_DIR" "${LOG_DIR}.$(date +%Y%m%d%H%M%S)"
fi

When debugging bash scripts, especially those involving remote SSH commands, we often need complete visibility into:

1. Command outputs (both stdout and stderr)
2. SSH connection status
3. Remote command execution results
4. Timing information between operations

Your current approach using | tee -a only captures explicit echo statements, missing critical execution details that could explain failures.

Here's an improved version of your script with full logging:

#!/bin/bash
LOGFILE="/home/scripts/cron/logs"

{
    # Print starting timestamp
    echo "=== SCRIPT START: $(date) ==="
    
    # Log part 1 with SSH status check
    echo "[$(date)] Part 1 - Starting..."
    if ssh_output=$(ssh -v -f admin@server.com 'bash /www/htdocs/server.com/scripts/part1.sh logout exit' 2>&1); then
        echo "[SUCCESS] Part 1 completed at $(date)"
        echo "SSH Output: $ssh_output"
    else
        echo "[FAILED] Part 1 at $(date)"
        echo "SSH Error: $ssh_output"
        exit 1
    fi
    
    # Sleep with logging
    echo "[$(date)] Sleeping for 120 seconds..."
    sleep 120
    
    # Subsequent parts with enhanced logging
    for part in {2..3}; do
        echo "[$(date)] Part $part - Starting..."
        if ssh_output=$(ssh -v admin@server.com "bash /www/htdocs/server.com/scripts/part${part}.sh logout exit" 2>&1); then
            echo "[SUCCESS] Part $part completed at $(date)"
            echo "SSH Output (truncated): ${ssh_output:0:500}..."
        else
            echo "[FAILED] Part $part at $(date)"
            echo "Full SSH Error: $ssh_output" >> "$LOGFILE.full"
            echo "See $LOGFILE.full for complete error details"
            exit 1
        fi
    done
    
    echo "=== SCRIPT END: $(date) ==="
} | tee -a "$LOGFILE"

Verbose SSH Logging (-v flag): The -v flag makes SSH output detailed connection information, which gets captured in our logs.

Error Handling: We now check SSH exit status explicitly and log success/failure states differently.

Output Capture: Using 2>&1 redirects stderr to stdout, and command substitution $(...) captures all output.

Debug Files: Large error outputs get written to separate .full files to prevent log flooding.

For even more comprehensive logging (including terminal control characters):

#!/bin/bash
LOGFILE="/home/scripts/cron/logs"

# Start session recording
script -q -a "$LOGFILE" -c '
    echo "Detailed logging started at $(date)"
    
    # Your commands here
    ssh -v admin@server.com "your_command"
    
    echo "Completed at $(date)"
'

# Compress the log (optional)
gzip "${LOGFILE}"

Log Rotation: Implement log rotation to prevent disk space issues from verbose logging.

Sensitive Data: Be cautious logging SSH sessions as they may contain passwords or keys (consider using SSH keys instead).

Performance Impact: Extensive logging adds overhead - balance between debug needs and production performance.