In Xen virtualized environments, it's common to see high CPU I/O wait percentages while disk activity appears minimal in tools like iostat. This occurs because:
# Typical iostat output showing the discrepancy
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
The root cause lies in Xen's split driver model:
- Frontend drivers in DomU (guest)
- Backend drivers in Dom0
- Event channels for communication
When the Dom0 scheduler is overloaded or network storage has latency spikes, the guest CPU will show I/O wait while actual disk metrics appear normal.
Use these Xen-specific commands to investigate:
# Check Xen scheduler stats
xentop -f
# Monitor event channel usage
xenstat -c 1
# Check block device backpressure
xl dmesg | grep blkback
Case 1: CPU Starvation in Dom0
When Dom0 doesn't get enough CPU cycles:
# Set Dom0 CPU allocation (in xl.cfg)
dom0_vcpus_pin="0-3"
dom0_mem=8192
Case 2: Network Storage Latency
For iSCSI/NFS backends showing as disk in guest:
# Monitor network latency
ping -c 10 storage-ip
netstat -s | grep retransmit
Case 3: Balloon Driver Contention
Memory pressure causing swap-like behavior:
# Check balloon driver
xl list -l | grep memory
xl balloon current-domain 8192
For deep analysis, use these techniques:
# Trace Xen hypercalls
xentrace -e 0x0003f000 -D /tmp/trace.dat
xentrace_format /usr/share/xentrace/formats /tmp/trace.dat
# Profile grant table usage
xl debug-keys g
When you observe significant CPU I/O wait percentages (like 50%) but minimal disk activity in iostat output, you're dealing with one of Linux system administration's more puzzling scenarios. The key indicators are:
$ iostat -x 1
avg-cpu: %user %nice %system %iowait %steal %idle
12.34 0.00 15.67 50.21 3.45 18.33
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz await svctm %util
xvda 0.00 0.00 0.20 0.10 5.00 2.00 40.00 1.50 1.00 0.03
In Xen virtualization environments, several unique factors can cause this behavior:
- Xen disk scheduler contention: The dom0's I/O scheduler competing with guest VMs
- Virtual block device throttling: Xen's blkback driver may be rate-limiting
- Memory ballooning pressure: When dom0 reclaims memory from guests
- XenStore latency: Delays in the control plane communication
To pinpoint the issue, try these Xen-specific commands:
# Check Xen block device stats xenstore-ls /local/domain/[domid]/device/vbd # Monitor grant table usage (critical for PV guests) xenstore-read /local/domain/[domid]/memory/grant-table/active-frames # Check for balloon driver activity xl list --long | grep memory
Even without NFS/FUSE, other network storage protocols might be involved:
# Check for iSCSI or AoE activity cat /proc/net/softnet_stat dmesg | grep -iE 'iscsi|aoe' # Alternative if top shows high CPU usage sudo perf top -g
Here's a script I've used to mitigate similar issues in production:
#!/bin/bash
# Tune Xen blkback parameters
echo 256 > /sys/module/xen_blkback/parameters/ring_pages
echo 1024 > /sys/module/xen_blkback/parameters/max_buffer_pages
# Adjust balloon driver
xl mem-set Domain-0 $(($(xl info | grep free_memory | awk '{print $3}') / 2))
# Change I/O scheduler (for dom0)
echo deadline > /sys/block/xvda/queue/scheduler
echo 64 > /sys/block/xvda/queue/nr_requests
When standard tools don't reveal the issue, deeper instrumentation helps:
# Install kernel debug symbols first sudo apt-get install linux-image-$(uname -r)-dbgsym # Trace blocking operations sudo perf record -e block:block_rq_issue -e block:block_rq_complete -a sleep 60 sudo perf script # Alternative with ftrace echo 1 > /sys/kernel/debug/tracing/events/block/enable cat /sys/kernel/debug/tracing/trace_pipe > block_trace.log