When dealing with vendor applications that generate massive log files (300MB/hour in this case) and refuse to implement proper logging controls, system administrators face several technical challenges:
- Continuous file handles maintained by the application
- Potential file locking issues during rotation
- Storage management with exponential growth
- Maintaining log accessibility for debugging
Here are proven approaches to handle aggressive log rotation on Windows:
1. Using Windows' Built-in Tools
The wevtutil
command can manage Windows Event Logs:
wevtutil sl Application /ms:10485760
wevtutil sl System /ms:10485760
2. Third-Party Rotation Utilities
LogRotateWin (GPL licensed):
# Sample configuration
C:\\vendor\\logs\\*.log {
size 10M
rotate 1000
daily
missingok
compress
delaycompress
sharedscripts
postrotate
net stop "VendorService" && net start "VendorService"
endscript
}
NXLog (Commercial with free edition):
define ROOT C:\\Program Files\\nxlog
define CERTDIR %ROOT%\\cert
<Extension _syslog>
Module xm_syslog
</Extension>
<Input in>
Module im_file
File "C:\\vendor\\logs\\app.log"
SavePos TRUE
Exec $Message = $raw_event;
</Input>
<Output out>
Module om_file
File "C:\\vendor\\logs\\app-%Y-%m-%d.log"
Exec if (file_size() > 10M) file_cycle();
</Output>
3. PowerShell Automation
For environments restricting third-party tools:
# LogRotate.ps1
$logPath = "C:\\vendor\\logs\\app.log"
$maxSize = 10MB
$maxFiles = 1000
$archivePattern = "app-{0:yyyyMMdd-HHmmss}.log" -f (Get-Date)
if ((Get-Item $logPath).Length -gt $maxSize) {
$archivePath = Join-Path (Split-Path $logPath) ($archivePattern)
Move-Item $logPath $archivePath -Force
# Optional: restart service if needed
# Restart-Service -Name "VendorService"
}
For applications that maintain persistent file handles:
- Shadow Copy Technique: Use VSS (Volume Shadow Copy) to create copies of active logs
- File System Minifilter: Develop a driver to intercept file operations (advanced)
- NTFS Hard Links: Create links while rotating the original file
# Create hard link example
fsutil hardlink create C:\\vendor\\logs\\app-archive.log C:\\vendor\\logs\\app.log
When implementing rotation for high-volume logs:
Method | Throughput | Application Impact |
---|---|---|
Scheduled Rotation | Medium | High (during rotation) |
Size-Based Rotation | High | Medium |
Filtered Logging | Highest | Lowest |
When dealing with vendor applications that generate massive log files (300MB/hour in this case) without built-in rotation features, Windows administrators face unique challenges. The core issue stems from Windows' lack of a native equivalent to Unix's logrotate utility, particularly when handling applications that maintain persistent file locks.
Here are proven approaches to implement log rotation on Windows:
1. Chomp Log Monitor (Active Fork)
- Download: http://chomp.kerys.co.uk/
- Config example:
<LogFile Name="AppLog.log">
<WhenFileExceeds Size="10MB">
<ZipMove/>
<DeleteFiles OlderThan="1d"/>
<KeepLast Count="1000"/>
</WhenFileExceeds>
</LogFile>
2. PowerShell Rotation Script
$logPath = "C:\Vendor\Logs\app.log"
$maxSize = 10MB
$retentionDays = 1
$maxFiles = 1000
if ((Get-Item $logPath).Length -gt $maxSize) {
$timestamp = Get-Date -Format "yyyyMMddHHmmss"
$newFile = "app_$timestamp.log"
Copy-Item $logPath "$(Split-Path $logPath)\$newFile"
Clear-Content $logPath
# Cleanup old files
Get-ChildItem "$(Split-Path $logPath)\app_*.log" |
Where-Object {$_.LastWriteTime -lt (Get-Date).AddDays(-$retentionDays)} |
Select-Object -First (Get-ChildItem "$(Split-Path $logPath)\app_*.log").Count -$maxFiles |
Remove-Item
}
When dealing with applications that maintain open handles:
- Use Volume Shadow Copy Service (VSS) for hot backups
- Consider NTFSSecurity PowerShell module for handle management
- For critical apps, schedule rotations during known low-activity periods
Tool | Key Feature | Handles Locked Files |
---|---|---|
LogRotateWin | logrotate port | Yes (with VSS) |
NXLog | Enterprise-grade | Yes |
Powershell Pro | Scripting framework | Partial |
When implementing any solution:
- Always test with a copy of production logs first
- Monitor file handles with Process Monitor during rotation
- Consider log shipping for centralized analysis before rotation
- Document rotation schedules and retention policies clearly