When dealing with long-running processes that output to log files, a common frustration occurs when using log rotation utilities like logrotate. The fundamental issue arises because:

1. The process maintains an open file descriptor to the original log file
2. logrotate moves/renames the original file (typical UNIX behavior)
3. The process continues writing to the original file descriptor
4. The new empty file created by logrotate remains unused

On Unix-like systems, when a process opens a file, it gets a file descriptor pointing to an inode (not a filename). When logrotate performs:

mv logfile.log logfile.log.1
touch logfile.log

The original file descriptor still points to the old inode (now named logfile.log.1), while the new logfile.log is a different inode entirely.

1. Using copytruncate in logrotate

Add this to your logrotate configuration:

/var/log/myapp.log {
    copytruncate
    daily
    rotate 7
    compress
    missingok
}

This works by:

1. Copying the original file
2. Truncating the original in place
3. Avoiding the move operation that breaks file descriptors

2. Process-Level Log Rotation

For applications you control, implement internal rotation:

# Python example
import logging
from logging.handlers import RotatingFileHandler

handler = RotatingFileHandler('app.log', maxBytes=1e6, backupCount=5)
logger = logging.getLogger()
logger.addHandler(handler)

3. Using socat for Persistent Redirection

A more advanced solution for arbitrary processes:

# Create a named pipe
mkfifo /var/log/mylog.fifo

# Redirect both stdout and stderr
your_command > /var/log/mylog.fifo 2>&1 &

# Use socat to handle the rotation
socat -u PIPE:/var/log/mylog.fifo \
      OPEN:/var/log/mylog.log,creat=0644,append \
      OPEN:/var/log/mylog.log.1,creat=0644,append=1

4. Systemd Services with StandardOutput

For systemd services, use:

[Service]
StandardOutput=append:/var/log/service.log
StandardError=inherit
ExecStart=/usr/bin/mycommand

Each solution has trade-offs:

• copytruncate can lose logs during rotation for high-volume applications
• Process-level rotation requires application modification
• Named pipes add complexity but provide the most reliable solution
• Systemd integration provides clean rotation but limits to systemd environments

For mission-critical applications, consider combining multiple approaches:

# logrotate config with postrotate signal
/var/log/critical.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    sharedscripts
    postrotate
        pkill -HUP mydaemon
    endscript
}

With application support for SIGHUP to reopen log files:

# Python signal handler example
import signal
import logging

def handle_hup(signum, frame):
    logger = logging.getLogger()
    for handler in logger.handlers[:]:
        handler.close()
        logger.removeHandler(handler)
    new_handler = logging.FileHandler('/var/log/critical.log')
    logger.addHandler(new_handler)

signal.signal(signal.SIGHUP, handle_hup)

---

When dealing with long-running processes that output logs to STDOUT, many developers face a frustrating scenario: after log rotation, new log files remain empty despite the process continuing to generate output. This occurs because traditional shell redirection (> or >>) maintains a file descriptor pointing to the original inode, not the filename.

Consider this common but problematic approach:

my_script.sh > /var/log/app.log

When logrotate moves app.log to app.log.1 and creates a new app.log, the process continues writing to the original file (now app.log.1) because it holds the original file descriptor.

1. Using a Logging Daemon

The most robust solution is to use a dedicated logging daemon:

# Using syslog
my_script.sh | logger -t myapp

# Using multilog from daemontools
my_script.sh | multilog t ./log

2. The nohup Approach

For simple cases where you want to maintain shell redirection:

nohup my_script.sh >> /var/log/app.log 2>&1 &

While this doesn't solve rotation by itself, it provides stability when combined with proper signal handling.

3. Custom Log Rotation Handling

For applications where you control the code, implement signal handling:

#!/bin/bash
# Setup log rotation signal handler
trap 'reopen_logs' SIGUSR1

logfile="/var/log/app.log"
exec 3>>"$logfile"

reopen_logs() {
    exec 3>&-  # Close current fd
    exec 3>>"$logfile"  # Reopen
}

# Main process output to fd 3
while true; do
    echo "Log entry $(date)" >&3
    sleep 1
done

4. Using FIFOs with Rotation

A more advanced solution using named pipes:

mkfifo /tmp/mylog.fifo
my_script.sh > /tmp/mylog.fifo &

# In another terminal or background process:
cat /tmp/mylog.fifo | tee -a /var/log/app.log

For mission-critical applications, consider these professional tools:

• logrotate with copytruncate option (though this has atomicity issues)
• Lumberjack or Filebeat for log shipping
• systemd-journald for system services