Best Log Rotation Tools and Techniques for Windows Servers Handling Large Vendor Logs (300MB+/Hour)


1 views

When dealing with vendor applications that generate massive log files (300MB/hour in this case) and refuse to implement proper logging controls, system administrators face several technical challenges:

  • Continuous file handles maintained by the application
  • Potential file locking issues during rotation
  • Storage management with exponential growth
  • Maintaining log accessibility for debugging

Here are proven approaches to handle aggressive log rotation on Windows:

1. Using Windows' Built-in Tools

The wevtutil command can manage Windows Event Logs:

wevtutil sl Application /ms:10485760
wevtutil sl System /ms:10485760

2. Third-Party Rotation Utilities

LogRotateWin (GPL licensed):

# Sample configuration
C:\\vendor\\logs\\*.log {
    size 10M
    rotate 1000
    daily
    missingok
    compress
    delaycompress
    sharedscripts
    postrotate
        net stop "VendorService" && net start "VendorService"
    endscript
}

NXLog (Commercial with free edition):

define ROOT C:\\Program Files\\nxlog
define CERTDIR %ROOT%\\cert

<Extension _syslog>
    Module xm_syslog
</Extension>

<Input in>
    Module im_file
    File "C:\\vendor\\logs\\app.log"
    SavePos TRUE
    Exec $Message = $raw_event;
</Input>

<Output out>
    Module om_file
    File "C:\\vendor\\logs\\app-%Y-%m-%d.log"
    Exec if (file_size() > 10M) file_cycle();
</Output>

3. PowerShell Automation

For environments restricting third-party tools:

# LogRotate.ps1
$logPath = "C:\\vendor\\logs\\app.log"
$maxSize = 10MB
$maxFiles = 1000
$archivePattern = "app-{0:yyyyMMdd-HHmmss}.log" -f (Get-Date)

if ((Get-Item $logPath).Length -gt $maxSize) {
    $archivePath = Join-Path (Split-Path $logPath) ($archivePattern)
    Move-Item $logPath $archivePath -Force
    # Optional: restart service if needed
    # Restart-Service -Name "VendorService"
}

For applications that maintain persistent file handles:

  • Shadow Copy Technique: Use VSS (Volume Shadow Copy) to create copies of active logs
  • File System Minifilter: Develop a driver to intercept file operations (advanced)
  • NTFS Hard Links: Create links while rotating the original file
# Create hard link example
fsutil hardlink create C:\\vendor\\logs\\app-archive.log C:\\vendor\\logs\\app.log

When implementing rotation for high-volume logs:

Method Throughput Application Impact
Scheduled Rotation Medium High (during rotation)
Size-Based Rotation High Medium
Filtered Logging Highest Lowest

When dealing with vendor applications that generate massive log files (300MB/hour in this case) without built-in rotation features, Windows administrators face unique challenges. The core issue stems from Windows' lack of a native equivalent to Unix's logrotate utility, particularly when handling applications that maintain persistent file locks.

Here are proven approaches to implement log rotation on Windows:

1. Chomp Log Monitor (Active Fork)
   - Download: http://chomp.kerys.co.uk/
   - Config example:
     <LogFile Name="AppLog.log">
       <WhenFileExceeds Size="10MB">
         <ZipMove/>
         <DeleteFiles OlderThan="1d"/>
         <KeepLast Count="1000"/>
       </WhenFileExceeds>
     </LogFile>

2. PowerShell Rotation Script
   $logPath = "C:\Vendor\Logs\app.log"
   $maxSize = 10MB
   $retentionDays = 1
   $maxFiles = 1000

   if ((Get-Item $logPath).Length -gt $maxSize) {
     $timestamp = Get-Date -Format "yyyyMMddHHmmss"
     $newFile = "app_$timestamp.log"
     Copy-Item $logPath "$(Split-Path $logPath)\$newFile"
     Clear-Content $logPath
     
     # Cleanup old files
     Get-ChildItem "$(Split-Path $logPath)\app_*.log" | 
       Where-Object {$_.LastWriteTime -lt (Get-Date).AddDays(-$retentionDays)} |
       Select-Object -First (Get-ChildItem "$(Split-Path $logPath)\app_*.log").Count -$maxFiles |
       Remove-Item
   }

When dealing with applications that maintain open handles:

  • Use Volume Shadow Copy Service (VSS) for hot backups
  • Consider NTFSSecurity PowerShell module for handle management
  • For critical apps, schedule rotations during known low-activity periods
Tool Key Feature Handles Locked Files
LogRotateWin logrotate port Yes (with VSS)
NXLog Enterprise-grade Yes
Powershell Pro Scripting framework Partial

When implementing any solution:

  1. Always test with a copy of production logs first
  2. Monitor file handles with Process Monitor during rotation
  3. Consider log shipping for centralized analysis before rotation
  4. Document rotation schedules and retention policies clearly