When working with Linux systems handling memory-intensive applications like MySQL, understanding swap behavior becomes crucial. The observed discrepancy between system-level swap reporting (via /proc/meminfo and top) and process-level swap accounting is a common yet poorly documented phenomenon.

# System-wide swap metrics (from /proc/meminfo)
SwapTotal: 2097148 kB     # Total swap space available
SwapFree:  865912 kB      # Currently unused swap
SwapCached: 24588 kB      # Pages swapped out but present in RAM

This indicates 1.23GB swap used (SwapTotal - SwapFree), while process-level reports show only ~440MB. The ~800MB difference warrants investigation.

The discrepancy stems from several technical aspects of Linux memory management:

# To examine swap areas and their utilization:
sudo swapon --show
NAME      TYPE      SIZE   USED PRIO
/dev/sda2 partition   2G 1.23G   -2

# Detailed swap usage by area:
sudo cat /proc/swaps
Filename        Type    Size    Used    Priority
/dev/sda2       partition   2097148 1231512 -2

1. Process-Swap Mapping

The original script provides useful but incomplete data. This enhanced version accounts for shared memory segments:

#!/bin/bash
for file in /proc/*/status ; do 
    awk '/Name|VmSwap|Shared|Pss/{printf $2 " " $3}END{ print ""}' $file 2>/dev/null 
done | sort -k3 -n -r | awk '{printf "%-30s %-20s %6d KB\n",$1,$2,$3}'

2. Examining Shared Memory Segments

# Check for shared memory segments using swap
sudo ipcs -m
sudo ipcs -m -p | awk '{print $3,$4,$6,$7}' | sort -n -k3

3. Kernel Slab Allocations

Kernel objects can consume swap space invisibly to userspace tools:

# Check slab cache usage
sudo cat /proc/meminfo | grep Slab
sudo cat /proc/slabinfo | awk '{if($3*$4/1024 > 1024) print $1,$3*$4/1024"KB"}'

Optimizing MySQL Memory Configuration

For database servers, improper memory allocation often leads to unexpected swap usage:

# Recommended my.cnf settings for 28GB RAM system:
[mysqld]
innodb_buffer_pool_size = 22G
innodb_log_file_size = 2G
innodb_flush_method = O_DIRECT
innodb_flush_neighbors = 0
innodb_read_io_threads = 16
innodb_write_io_threads = 16

Using smem for Accurate Reporting

# Install and run smem for comprehensive reporting
sudo apt-get install smem
sudo smem -t -k -s swap -n -c "name pss swap"

Kernel Page Table Examination

For deep forensic analysis (requires kernel debugging):

# Requires debug symbols installed
sudo perf probe -a 'shmem_swapin_fault'
sudo perf stat -e 'probe:shmem_swapin_fault' -a sleep 10

Understanding these mechanisms helps properly tune systems and identify when swap usage becomes problematic versus when it's simply efficient memory management at work.

---

When monitoring swap usage on Linux systems, administrators often notice discrepancies between system-wide swap metrics (from /proc/meminfo or top) and process-level swap accounting. This becomes particularly apparent in database servers like MySQL where memory management is critical.

Here are the key metrics from the example scenario:

# System-wide view:
SwapTotal: 2097148 kB
SwapFree:  865912 kB
SwapUsed:  1231512 kB

# Process-level view (from script):
Total Swap Used: ~449264 kB

The 800MB+ discrepancy suggests there's more to swap accounting than simple process-level summation.

Linux swap behavior involves several complex mechanisms:

1. Shared memory pages in swap
2. Swap cache (SwapCached)
3. Anonymous memory that's been swapped out
4. Memory that was swapped but subsequently read back

Here are more accurate ways to inspect swap usage:

Method 1: Using smem

smem provides better swap accounting:

sudo smem -s swap -r | head -20

Method 2: Enhanced Swap Script

An improved version of the process swap script:

#!/bin/bash
for file in /proc/*/status ; do 
    awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file 2>/dev/null 
done | sort -k 2 -n -r | less

Method 3: Examining Page Tables

For advanced diagnosis, examine page tables:

sudo grep -i swap /proc/*/smaps 2>/dev/null | 
    awk '{split($0,a,"/"); printf("%s %s\n",a[3],$0)}' | 
    sort -k1,1

Database systems like MySQL exhibit special swap behavior:

# Check InnoDB buffer pool status:
SHOW ENGINE INNODB STATUS\G

# Monitor memory usage:
SELECT * FROM sys.memory_global_by_current_bytes 
WHERE event_name LIKE 'memory/innodb%';

The missing swap usage typically comes from:

1. Shared library memory swapped out
2. tmpfs pages that have been swapped
3. Kernel slab allocations
4. Memory that was swapped but the process terminated
5. Swap used by kernel threads

To identify these specifically:

# Check for swapped tmpfs:
sudo find /proc/*/smaps -exec grep -i swap {} + 2>/dev/null

# Check slab info:
sudo cat /proc/slabinfo | grep -v ^# | sort -k2 -n -r

For production MySQL servers consider:

# Immediate relief:
sudo swapoff -a && sudo swapon -a

# Persistent configuration:
echo 'vm.swappiness = 5' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

# MySQL-specific tuning:
SET GLOBAL innodb_buffer_pool_dump_at_shutdown=ON;
SET GLOBAL innodb_buffer_pool_load_at_startup=ON;

Create a monitoring script:

#!/bin/bash
date
echo "System swap:"
free -h | grep -i swap
echo "Process swap:"
sudo smem -t -k -s swap -c "name swap" | grep -v "0K"