When analyzing your Apache processes, several red flags emerge:

PID    USER     PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ COMMAND
20839 www-data  20  0  1008m 359m 22m S  4    4.8    1:52.61 apache2

The key metrics showing potential memory issues:

• RES (resident memory) consistently above 350MB per process
• Process lifetime exceeding normal request duration (1-2 hours)
• Strace showing restart_syscall as dominant system call

The strace output:

restart_syscall(<... resuming interrupted call ...> 

This indicates the process was:

1. Interrupted by a signal during a system call
2. Automatically restarted the system call after signal handling
3. Currently waiting in an incomplete system call

Common scenarios where this occurs:

// Example of code that might trigger this pattern
while (1) {
    // Blocking I/O operation
    bytes_read = read(fd, buffer, sizeof(buffer));  // System call that can be interrupted
    if (bytes_read == -1 && errno == EINTR) {
        continue;  // Common pattern for signal handling
    }
    // Process data
}

To properly diagnose the memory growth:

# Install debug symbols (Ubuntu/Debian)
sudo apt-get install apache2-dbg

# Attach gdb to running process
sudo gdb -p 20839

# In gdb:
(gdb) info proc mappings
(gdb) malloc_info 0 stdout

Alternative tools for memory analysis:

• valgrind --leak-check=full --show-leak-kinds=all
• pmap -x [PID] for memory segment breakdown
• Apache's mod_status for request tracking

While investigating root cause, implement these mitigations:

# httpd.conf or apache2.conf adjustments

    StartServers            5
    MinSpareServers         5
    MaxSpareServers         10
    MaxRequestWorkers       150  # Previously MaxClients
    MaxConnectionsPerChild  1000 # Previously MaxRequestsPerChild

Additional PHP-specific settings if applicable:

; php.ini modifications
memory_limit = 128M
max_execution_time = 30
opcache.memory_consumption = 64

Implement these tracking mechanisms:

#!/bin/bash
# Monitor Apache memory usage
while true; do
    date >> /var/log/apache_mem.log
    ps -eo pid,user,comm,rss,etime | grep apache2 >> /var/log/apache_mem.log
    free -m >> /var/log/apache_mem.log
    sleep 300
done

Consider implementing:

• Prometheus + Grafana for visualization
• APM tools like New Relic or Datadog
• Custom metric collection using mod_status

For deeper analysis of system calls:

# SystemTap script to trace memory allocation
probe process("apache2").function("apr_palloc") {
    printf("%s size: %d\n", pp(), $size)
}
probe process("apache2").function("apr_palloc").return {
    printf("allocated at %p\n", $return)
}

This helps identify:

• Memory allocation patterns
• Potential leak locations
• Request processing bottlenecks

---

When you see restart_syscall in strace output from an Apache process, it typically indicates the process was interrupted (often by a signal) and is now resuming its system call. While this message itself isn't problematic, combined with your memory usage patterns, it suggests deeper resource management issues.

# Example strace output showing interrupted call
$ sudo strace -p 20839
restart_syscall(<... resuming interrupted call ...> <unfinished ...>

Your Apache processes showing consistent large memory usage (~350MB+) and long uptimes are classic memory leak symptoms. Key observations:

• Processes maintaining high RSS (resident set size) over time
• No natural memory release between requests
• Daily restarts required to prevent OOM (Out of Memory) situations

Lowering MaxRequestsPerChild is indeed a recommended approach:

# Recommended configuration in apache2.conf or httpd.conf
<IfModule prefork.c>
  StartServers        5
  MinSpareServers     5
  MaxSpareServers    10
  MaxClients        150
  MaxRequestsPerChild 1000  # Reduced from default 0 (unlimited)
</IfModule>

The optimal value depends on your leak rate. Start with 1000 and monitor:

# Bash script to monitor Apache memory usage
#!/bin/bash
watch -n 60 "ps -ylC apache2 --sort:rss | awk '{print \$8,\$9,\$10,\$2}'"

For deeper analysis, combine strace with memory profiling:

# Detailed strace capturing memory-related calls
sudo strace -p 20839 -e trace=mmap,munmap,brk,read,write -f -s 256 -o /tmp/apache_trace.log

# Check PHP memory leaks (if using mod_php)
sudo apt-get install php-memprof
# Add to your PHP script:
memprof_enable();
// Your code
memprof_dump_callgrind(fopen("output.callgrind", "w"));

Consider switching to mpm_event for better memory management:

# On Debian/Ubuntu:
sudo a2dismod mpm_prefork
sudo a2enmod mpm_event
sudo systemctl restart apache2

# Sample event MPM configuration
<IfModule mpm_event_module>
  ServerLimit             16
  StartServers            2
  MinSpareThreads        25
  MaxSpareThreads        75
  ThreadLimit            64
  ThreadsPerChild        25
  MaxRequestWorkers     400
  MaxConnectionsPerChild 10000
</IfModule>

Implement a monitoring system to track memory usage patterns:

# Sample Prometheus query for Apache memory
process_resident_memory_bytes{job="apache"} > 300000000
# Alert rule example
groups:
- name: apache.rules
  rules:
  - alert: ApacheMemoryLeak
    expr: increase(process_resident_memory_bytes{job="apache"}[1h]) > 100000000
    for: 15m