When your EC2 instances periodically freeze with Munin graphs showing Apache processes consuming increasing memory until swap exhaustion, you're likely dealing with a classic memory leak. The key indicators:
# Sample munin memory graph pattern:
1. Steady baseline memory usage
2. Sudden upward trend in "apps" memory
3. Continuous growth until swap fills
4. System becomes unresponsive
Before diving deep, establish these monitoring basics:
# Install essential tools
sudo apt-get install strace htop
# Monitor Apache processes
watch -n 1 "ps aux | grep apache | sort -nk 4"
# Track memory allocation
sudo strace -p [PID] -tt -o trace.log -s 256 -e brk
The breakthrough came when correlating strace output with PHP execution points:
// Add debug points in PHP
error_log("MEMCHECK: " . shell_exec("ps -p " . getmypid() . " -o rss="));
// Sample strace analysis:
$ grep -A 5 "write(2, \"MEMCHECK" trace.log
$ grep -B 10 "brk(0x" trace.log | tail -20
The smoking gun appeared in the SSL handshake:
// Problematic code:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://api.example.com");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch); // Leaks ~400KB per call
curl_close($ch);
// Working alternative:
curl_setopt($ch, CURLOPT_URL, "http://api.example.com"); // No leak
These config changes bought time for proper fixes:
# Apache config (prefork)
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 50 # Critical for leak containment
The complete resolution involved multiple steps:
# Upgrade libcurl (Ubuntu example)
sudo apt-get install libcurl4-openssl-dev
# Verify version
php -r "echo curl_version()['version'];"
// PHP code improvement
$ch = curl_init();
if (strpos($url, 'https') === 0) {
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // Temporary workaround
}
// Always clean up
register_shutdown_function(function() use ($ch) {
@curl_close($ch);
});
Implement these checks to catch future leaks early:
# Cron job to monitor Apache memory
*/5 * * * * /usr/bin/php /var/www/scripts/memcheck.php
# memcheck.php content:
$threshold = 500; // MB
$memory = shell_exec("ps -C apache2 -o rss= | awk '{sum+=$1} END {print sum/1024}'");
if ($memory > $threshold) {
mail('admin@example.com', 'Memory Alert', "Apache using {$memory}MB");
}
When dealing with memory leaks in Apache/PHP environments, the first challenge is identifying the pattern. In my case, EC2 instances became unresponsive periodically, with Munin graphs showing:
- Gradual memory consumption growth in "apps" category
- Apache processes consuming memory without releasing it
- Complete swap exhaustion leading to system crashes
Before jumping to conclusions about PHP or Apache being the culprit, I implemented this diagnostic strategy:
# Monitor Apache processes sorted by memory usage
watch -n 5 "ps -ylC apache2 --sort:rss | head -20"
The breakthrough came when using strace to track memory allocations at the system call level:
# Attaching to a suspicious Apache process
strace -p PID -tt -o trace.log -s 256
Key findings in the output:
- Repeated brk() calls showing heap expansion
- 400KB allocations occurring per request
- Correlation with specific PHP function calls
Through systematic tracing, the leak was traced to cURL's SSL handling:
// Problematic code pattern
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://example.com");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch); // Memory leak here
curl_close($ch);
While working on a permanent fix, these measures helped stabilize the system:
# Apache configuration adjustments
<IfModule prefork.c>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 50
MaxRequestsPerChild 100 # Forces process recycling
</IfModule>
The complete resolution involved multiple steps:
- Upgrading cURL to version 7.19.5+ (fixed known SSL leaks)
- Implementing proper resource cleanup:
// Safe cURL usage pattern $ch = curl_init(); try { // ... configuration ... $response = curl_exec($ch); if ($response === false) { throw new RuntimeException(curl_error($ch)); } return $response; } finally { curl_close($ch); unset($ch); // Explicit cleanup }
- Adding memory monitoring to critical operations