When debugging bash scripts, especially those involving remote commands via SSH, we often face the challenge of incomplete logging. The current approach using tee
only captures explicitly echoed messages, missing crucial SSH output and potential error messages.
Bash handles three standard streams:
1. stdin (0) - Standard input
2. stdout (1) - Standard output
3. stderr (2) - Standard error
Your current solution only captures stdout from echo commands. SSH command outputs and errors go to different streams.
Here's an improved version that captures everything:
#!/bin/bash
{
echo " $(date) : part 1 - start "
ssh -f admin@server.com 'bash /www/htdocs/server.com/scripts/part1.sh logout exit'
echo " $(date) : sleep 120"
sleep 120
echo " $(date) : part 2 - start"
ssh admin@server.com 'bash /www/htdocs/server.com/scripts/part2.sh logout exit'
echo " $(date) : part 3 - start"
ssh admin@server.com 'bash /www/htdocs/server.com/scripts/part3.sh logout exit'
echo " $(date) : END"
} 2>&1 | tee -a /home/scripts/cron/logs
2>&1
redirects stderr (2) to stdout (1), ensuring error messages are captured. The entire script block is wrapped in curly braces to apply the redirection to all commands.
For more complex scripts, consider using exec
:
#!/bin/bash
log_file="/home/scripts/cron/logs"
exec > >(tee -a "${log_file}") 2>&1
echo " $(date) : part 1 - start "
ssh -f admin@server.com 'bash /www/htdocs/server.com/scripts/part1.sh logout exit'
# Rest of your script...
To verify SSH command success in your logs, use return code checks:
if ! ssh admin@server.com 'bash /path/to/script.sh'; then
echo " $(date) : ERROR - SSH command failed"
exit 1
fi
For better debugging, add millisecond precision timestamps:
#!/bin/bash
log_with_timestamp() {
while IFS= read -r line; do
printf '%(%Y-%m-%d %H:%M:%S)T %s\n' -1 "$line"
done
}
{
# Your commands here
} 2>&1 | log_with_timestamp | tee -a /home/scripts/cron/logs
For production systems, implement log rotation:
#!/bin/bash
LOG_DIR="/home/scripts/cron/logs"
MAX_LOG_SIZE=10485760 # 10MB
if [ -f "$LOG_DIR" ] && [ $(stat -c%s "$LOG_DIR") -gt $MAX_LOG_SIZE ]; then
mv "$LOG_DIR" "${LOG_DIR}.$(date +%Y%m%d%H%M%S)"
fi
When debugging bash scripts, especially those involving remote SSH commands, we often need complete visibility into:
1. Command outputs (both stdout and stderr)
2. SSH connection status
3. Remote command execution results
4. Timing information between operations
Your current approach using | tee -a
only captures explicit echo
statements, missing critical execution details that could explain failures.
Here's an improved version of your script with full logging:
#!/bin/bash
LOGFILE="/home/scripts/cron/logs"
{
# Print starting timestamp
echo "=== SCRIPT START: $(date) ==="
# Log part 1 with SSH status check
echo "[$(date)] Part 1 - Starting..."
if ssh_output=$(ssh -v -f admin@server.com 'bash /www/htdocs/server.com/scripts/part1.sh logout exit' 2>&1); then
echo "[SUCCESS] Part 1 completed at $(date)"
echo "SSH Output: $ssh_output"
else
echo "[FAILED] Part 1 at $(date)"
echo "SSH Error: $ssh_output"
exit 1
fi
# Sleep with logging
echo "[$(date)] Sleeping for 120 seconds..."
sleep 120
# Subsequent parts with enhanced logging
for part in {2..3}; do
echo "[$(date)] Part $part - Starting..."
if ssh_output=$(ssh -v admin@server.com "bash /www/htdocs/server.com/scripts/part${part}.sh logout exit" 2>&1); then
echo "[SUCCESS] Part $part completed at $(date)"
echo "SSH Output (truncated): ${ssh_output:0:500}..."
else
echo "[FAILED] Part $part at $(date)"
echo "Full SSH Error: $ssh_output" >> "$LOGFILE.full"
echo "See $LOGFILE.full for complete error details"
exit 1
fi
done
echo "=== SCRIPT END: $(date) ==="
} | tee -a "$LOGFILE"
Verbose SSH Logging (-v flag): The -v
flag makes SSH output detailed connection information, which gets captured in our logs.
Error Handling: We now check SSH exit status explicitly and log success/failure states differently.
Output Capture: Using 2>&1
redirects stderr to stdout, and command substitution $(...)
captures all output.
Debug Files: Large error outputs get written to separate .full
files to prevent log flooding.
For even more comprehensive logging (including terminal control characters):
#!/bin/bash
LOGFILE="/home/scripts/cron/logs"
# Start session recording
script -q -a "$LOGFILE" -c '
echo "Detailed logging started at $(date)"
# Your commands here
ssh -v admin@server.com "your_command"
echo "Completed at $(date)"
'
# Compress the log (optional)
gzip "${LOGFILE}"
Log Rotation: Implement log rotation to prevent disk space issues from verbose logging.
Sensitive Data: Be cautious logging SSH sessions as they may contain passwords or keys (consider using SSH keys instead).
Performance Impact: Extensive logging adds overhead - balance between debug needs and production performance.