When running long-lived processes with nohup, the output file can grow uncontrollably, consuming disk space. Many developers face this issue but hesitate to rotate logs because restarting critical applications isn't always an option.

The key lies in how Linux handles file descriptors. When a process writes to nohup.out, it maintains an open file descriptor pointing to the inode, not the filename. This means we can:

1. Rename the existing file
2. Create a new empty file with the same name
3. Optionally compress/archive the old logs

Here's the step-by-step approach:

# First, rename the existing file
mv nohup.out nohup.out.old

# Create new empty file
touch nohup.out

# If you need to keep the old logs compressed
gzip nohup.out.old

For automated rotation, create a /etc/logrotate.d/nohup configuration:

/path/to/your/nohup.out {
    daily
    rotate 7
    compress
    missingok
    notifempty
    copytruncate
    size 100M
}

The copytruncate option is crucial - it copies the file then truncates it rather than moving and recreating.

A more elegant solution is to avoid nohup.out altogether:

nohup your_command > /var/log/your_app.log 2>&1 &

Then you can manage /var/log/your_app.log with standard log rotation tools.

For multiple instances writing to the same file, consider this pattern:

nohup your_command > /var/log/your_app_$(date +%Y%m%d_%H%M%S).log 2>&1 &

After rotation, verify the process is still writing to the correct file:

lsof -p PID | grep nohup.out

This will show the file descriptor information for your process.

---

When running long-lived processes with nohup, the nohup.out file can quickly become massive. Unlike regular log files managed by logging frameworks, nohup.out doesn't support built-in rotation. This creates two main issues:

• Disk space consumption
• Difficulty in parsing historical logs

Common approaches like logrotate or manual mv operations don't work well because:

# This approach breaks the application's output stream
mv nohup.out nohup.out.1
touch nohup.out  # The application still writes to the old file descriptor

Option 1: Redirect to a Proper Logging System

The most robust solution is to avoid nohup.out entirely:

nohup your_application > /var/log/your_app.log 2>&1 &

Then configure proper log rotation in /etc/logrotate.d/your_app:

/var/log/your_app.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
    copytruncate
}

Option 2: Use a Pipe with Rotation

For cases where you must keep using nohup:

# Create a named pipe
mkfifo /tmp/log_pipe

# Start a rotation-aware consumer
( while true; do
    current_date=$(date +%Y%m%d)
    cat /tmp/log_pipe >> /var/log/app_${current_date}.log
  done
) &

# Launch your application
nohup your_application > /tmp/log_pipe 2>&1 &

Option 3: The Copy-Truncate Method

If you must rotate an existing nohup.out:

# First, copy the current file
cp --attributes-only --preserve=all nohup.out nohup.out.1

# Then truncate the original
: > nohup.out

Note: This method may lose some log entries during the operation.

• Use proper logging frameworks (Log4j, Winston, etc.) instead of stdout
• Implement log rotation at the application level
• For critical systems, consider centralized logging solutions

After implementing rotation, verify it's working with:

# Check file descriptors
ls -l /proc/$(pgrep your_application)/fd