How to Append Cron Job Output to Log File Instead of Overwriting


1 views

When setting up cron jobs, a common pitfall is using the single > operator for output redirection, which overwrites the log file on each execution. Here's what most beginners use:

*/10 * * * * /path/to/script.sh > /var/log/cron.log 2>&1

To preserve previous logs while adding new entries, simply replace > with >> (double greater-than symbols):

*/10 * * * * /path/to/script.sh >> /var/log/cron.log 2>&1

The >> operator appends output to existing files rather than truncating them.

For the specific case mentioned in the question with a Python script, here's the corrected version:

*/10 * * * * /usr/local/bin/python2.7 ~/webapps/****/WR/cron.py >> ~/webapps/****/WR/cron.log 2>&1

While appending solves the immediate problem, consider these additional improvements for production environments:

# Create dated log files
*/10 * * * * /usr/local/bin/python2.7 ~/webapps/****/WR/cron.py >> ~/webapps/****/WR/cron-$(date +\%Y-\%m-\%d).log 2>&1

# Or implement log rotation
*/10 * * * * /usr/local/bin/python2.7 ~/webapps/****/WR/cron.py >> ~/webapps/****/WR/cron.log 2>&1
0 0 * * * mv ~/webapps/****/WR/cron.log ~/webapps/****/WR/cron.log.$(date +\%Y\%m\%d)

For system-wide logging, you can pipe output to the system logger:

*/10 * * * * /usr/local/bin/python2.7 ~/webapps/****/WR/cron.py 2>&1 | logger -t mycron

When setting up cron jobs, many developers encounter the frustrating behavior where output redirection (> file.log) overwrites the previous log file contents. This becomes problematic when you need historical data for debugging or monitoring purposes.

The fundamental difference between these operators explains the behavior:

> file.log   # Overwrites existing file
>> file.log  # Appends to existing file

For the specific cron job example mentioned:

*/10 * * * * /usr/local/bin/python2.7 ~/webapps/****/WR/cron.py >> ~/webapps/****/WR/cron.log 2>&1

Key changes:

  • Replaced single > with >>
  • Maintained the error redirection (2>&1)
  • Kept all other parameters identical

For long-running cron jobs, consider implementing log rotation:

# Daily log rotation example
0 0 * * * mv ~/webapps/****/WR/cron.log ~/webapps/****/WR/cron-$(date +\%Y-\%m-\%d).log

For system-level logging:

*/10 * * * * /usr/local/bin/python2.7 ~/webapps/****/WR/cron.py | logger -t mycronjob

If changes don't take effect:

  • Verify cron service is running: service cron status
  • Check permissions on log file directory
  • Test command manually first