The Linux Out-of-Memory (OOM) killer activates when the system faces severe memory pressure, even when swap space exists. From your logs, we can see the process memcheck-amd64- was killed despite having 3GB of free swap space.

Mar  7 02:43:11 myhost kernel: Out of memory: Kill process 3841 (memcheck-amd64-) score 11 or sacrifice child
Mar  7 02:43:11 myhost kernel: Killed process 3841 (memcheck-amd64-) total-vm:128024kB, anon-rss:0kB, file-rss:0kB

1. Adjusting OOM Score for Critical Processes

The most precise method is modifying the OOM score adjustment value for specific processes:

# Make the process extremely unlikely to be killed
echo -1000 > /proc/[PID]/oom_score_adj

# Check current score
cat /proc/[PID]/oom_score

For persistent configuration, create a systemd service or startup script that applies this to your process automatically.

2. Memory Limits and Control Groups

Create a dedicated cgroup with memory guarantees:

# Create a new cgroup
cgcreate -g memory:valgrind_group

# Set memory limits (200MB in your case)
cgset -r memory.limit_in_bytes=200M valgrind_group

# Launch process in the cgroup
cgexec -g memory:valgrind_group valgrind [options] your_program

3. Kernel Parameter Tweaks

Additional sysctl parameters that influence OOM behavior:

# Make kernel prefer swapping over killing processes
echo 60 > /proc/sys/vm/swappiness

# Protect specific process types (like SSH)
echo 'vm.oom_kill_allocating_task = 0' >> /etc/sysctl.conf
sysctl -p

For your valgrind testing scenario, consider these specialized approaches:

# Use --vgdb=no to reduce valgrind's memory footprint
valgrind --vgdb=no --leak-check=full your_program

# Alternatively, limit valgrind's own memory usage
export VALGRIND_OPTS="--freelist-vol=20000000 --freelist-big-blocks=1000000"

Create a monitoring script to watch your protected process:

#!/bin/bash
PID=$(pgrep -f "memcheck-amd64-")
while true; do
    echo "OOM Score: $(cat /proc/$PID/oom_score)"
    echo "Memory Usage: $(pmap -x $PID | tail -1)"
    sleep 5
done

Combine these techniques for robust protection during your memory-intensive testing.

The Linux Out-of-Memory (OOM) killer is designed to terminate processes when the system runs critically low on memory. However, as seen in your case, it may trigger prematurely even when swap space is available. The kernel's decision is based on complex heuristics involving:

• oom_score calculation
• memory pressure thresholds
• process characteristics

Here are the most effective ways to modify OOM killer behavior:

# Temporary adjustment for a running process
echo -17 > /proc/[PID]/oom_score_adj

# Permanent solution via sysctl
sudo sysctl -w vm.overcommit_memory=2
sudo sysctl -w vm.overcommit_ratio=80
sudo sysctl -w vm.swappiness=60

For critical processes like your valgrind testing:

# Make a process virtually immune to OOM killer
sudo choom -p [PID] -n -1000

# Alternative method using cgroups
sudo cgcreate -g memory:my_protected_group
echo "-1000" > /sys/fs/cgroup/memory/my_protected_group/memory.oom_control
echo [PID] > /sys/fs/cgroup/memory/my_protected_group/tasks

Add these to /etc/sysctl.conf for persistent changes:

vm.panic_on_oom=0
vm.oom_kill_allocating_task=0
vm.overcommit_memory=2
vm.overcommit_ratio=80

Use these commands to monitor memory pressure:

# Check current OOM scores
ps -eo pid,comm,oom_score,oom_score_adj | sort -k3 -n

# Monitor swap usage
vmstat 1 5

# Detailed memory info
cat /proc/meminfo | grep -E 'Mem|Swap'

Create a script to monitor and prevent OOM situations:

#!/bin/bash
THRESHOLD=90
while true; do
  MEM_USED=$(free | awk '/Mem/{printf("%d"), $3/$2*100}')
  if [ "$MEM_USED" -ge "$THRESHOLD" ]; then
    echo "Warning: Memory usage at $MEM_USED%" | mail -s "Memory Alert" admin@example.com
    # Optionally trigger memory cleanup
  fi
  sleep 60
done

When running memory-intensive tools like valgrind:

# Launch with adjusted OOM score
exec /usr/bin/choom -n -500 -- /usr/bin/valgrind [options] your_program

# Alternative: Use cgroups with memory limits
sudo cgcreate -g memory:valgrind_group
echo "4G" > /sys/fs/cgroup/memory/valgrind_group/memory.limit_in_bytes
echo [PID] > /sys/fs/cgroup/memory/valgrind_group/tasks