When monitoring our Bacula backup server, I noticed severe performance degradation despite having 40% of physical memory (RAM) available. The system was aggressively using swap space (1.8GB out of 7.6GB) while showing 1.6GB free RAM in free -m output:

free -m
             total       used       free     shared    buffers     cached
Mem:          3949       2356       1593          0          0          1
-/+ buffers/cache:       2354       1595
Swap:         7629       1804       5824

The vmstat 1 output reveals heavy disk I/O (wa time 20-30%) and significant swap activity:

procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  2 1843536 1634512      0   4188   54   13  2524   666    2    1  1  1 89  9  0
 1 11 1845916 1640724      0    388 2700 4816 221880  4879 14409 170721  4  3 63 30  0

From the top output sorted by CPU time:

• kswapd0 (kernel swap daemon) is consuming significant CPU (3% with 831 minutes)
• Bacula processes (bacula-fd, bacula-sd) are active but not memory-intensive
• High I/O wait times (wa) between 20-35%

The issue stems from Linux's swappiness behavior. Even with available RAM, Linux may swap when:

1. vm.swappiness is set too high (default is 60)
2. Memory pressure exists from inactive pages
3. Filesystem cache isn't being reclaimed properly

Check current swappiness value:

cat /proc/sys/vm/swappiness

Immediate fix - Reduce swappiness:

sudo sysctl vm.swappiness=10

To make permanent, add to /etc/sysctl.conf:

vm.swappiness = 10

For Bacula-specific tuning:

# In bacula-dir.conf
Maximum Concurrent Jobs = 5
# Reduce if needed based on your hardware

Monitor memory usage by process:

sudo smem -s swap -r

Check which files are swapped out:

sudo find /proc -maxdepth 2 -path "/proc/[0-9]*/status" -readable \
-exec awk -v PID="$(basename $(dirname {}))" \
'{if($1=="VmSwap:" && $2>0) print $2 " kB swapped for PID: " PID " - " $3}' {} \;

• Implement proper monitoring with alerts for swap usage
• Consider adding more RAM if backups continue growing
• Schedule large backups during off-peak hours
• For critical systems, consider setting swappiness=1 or 0

---

The server shows an unusual memory pattern where it's actively swapping despite having 40% free physical memory. From the free -m output:

Mem:          3949       2356       1593          0          0          1
-/+ buffers/cache:       2354       1595
Swap:         7629       1804       5824

This indicates 1.6GB free memory but 1.8GB swap usage - a clear red flag for performance issues.

The vmstat 1 output reveals heavy I/O wait (wa) and swap activity:

procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  2 1843536 1634512      0   4188   54   13  2524   666    2    1  1  1 89  9  0
 1 11 1845916 1640724      0    388 2700 4816 221880  4879 14409 170721  4  3 63 30  0

Key observations:

• High swap-in (si) and swap-out (so) values
• I/O wait (wa) consistently above 20%
• Block I/O operations (bi/bo) in thousands

The top output shows kswapd0 consuming significant CPU time:

 240 root      15   0     0    0    0 D    3  0.0 831:55.19 831:55 kswapd0

This kernel process is responsible for managing swap operations and shouldn't normally consume this much CPU.

Several factors could contribute to this behavior:

# Check current swappiness value
cat /proc/sys/vm/swappiness

# Verify memory pressure settings
cat /proc/sys/vm/vfs_cache_pressure

Try these commands to temporarily alleviate the issue:

# Reduce swappiness immediately
echo 10 > /proc/sys/vm/swappiness

# Clear page cache (careful in production)
echo 3 > /proc/sys/vm/drop_caches

# Check for memory leaks in running processes
ps aux --sort=-%mem | head -n 10

Consider these configuration changes in /etc/sysctl.conf:

vm.swappiness = 10
vm.vfs_cache_pressure = 50
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

Apply changes with:

sysctl -p

Implement better monitoring with these commands:

# Install if not available
apt-get install sysstat

# Continuous memory monitoring
sar -r 1 3

# Detailed process memory usage
pmap -x [PID]

For the Bacula backup system mentioned, consider these adjustments:

# In bacula-dir.conf
Maximum Concurrent Jobs = 5
Maximum Concurrent Storage Jobs = 3

# In bacula-sd.conf
Maximum Concurrent Jobs = 3
Maximum Concurrent Read Jobs = 2