When analyzing disk performance on Linux systems, two metrics often cause confusion:
- IOPS (Input/Output Operations Per Second): Measures the number of distinct read/write operations
- TPS (Transfers Per Second) from
sar
/iostat
: Counts the number of transactions between the device driver and storage
Device: tps Blk_read/s Blk_wrtn/s
sda 559.00 0.00 142600.00
dm-0 18433.00 0.00 147464.00
The high TPS values (especially for dm devices) occur because:
- A single logical I/O can generate multiple physical transactions
- Device mapper layers (like LVM) split operations further
- Kernel block layer merges/splits requests
For accurate IOPS measurement:
# Measure random read IOPS (4KB blocks)
fio --name=random-read --ioengine=libaio --rw=randread \
--bs=4k --direct=1 --size=1G --numjobs=1 \
--runtime=30 --time_based --end_fsync=1
# Measure sequential write throughput
fio --name=seq-write --ioengine=libaio --rw=write \
--bs=128k --direct=1 --size=1G --numjobs=1 \
--runtime=30 --time_based --end_fsync=1
Metric | What It Measures | Typical Values |
---|---|---|
tps | Transactions per second | Depends on I/O pattern |
Blk_read/s | Blocks read per second | 512-byte blocks |
Blk_wrtn/s | Blocks written per second | 512-byte blocks |
%iowait | CPU waiting for I/O | 5-10% is normal |
Several factors affect actual disk performance:
- Queue depth: Higher depths enable more parallel operations
# Check queue depth cat /sys/block/sda/queue/nr_requests
- Filesystem overhead: EXT4 vs XFS vs Btrfs
- I/O scheduler: Deadline, CFQ, or NOOP
# Check current scheduler cat /sys/block/sda/queue/scheduler
For production systems:
- Use
fio
for realistic benchmarks - Monitor
await
andsvctm
iniostat -x
- Consider filesystem journaling impact
- Test with production-like workload patterns
When examining disk performance metrics in Linux systems, it's crucial to understand that IOPS (Input/Output Operations Per Second) and tps (Transfers Per Second) represent different concepts despite their apparent similarity.
IOPS measures the raw number of read/write operations the storage device can handle, while tps (as reported by tools like sar
and iostat
) counts logical I/O requests issued to the device driver:
# Conceptual breakdown:
Physical IOPS (hardware capability) ≤ Device tps (kernel requests) ≤ Filesystem IOPS (VFS layer)
The discrepancy you're observing stems from several architectural factors:
- Block Device Layer Abstraction: The kernel merges adjacent requests and splits large ones
- Filesystem Overhead: Journaling, metadata operations, and write barriers create additional transactions
- Device Mapper Impact: Your
dm-*
devices show how LVM/RAID layers multiply underlying I/O
For accurate IOPS measurement, combine these tools:
# Raw device testing (bypass filesystem cache)
fio --name=randread --ioengine=libaio --rw=randread --bs=4k \
--direct=1 --size=1G --numjobs=1 --runtime=60 --time_based
# Correlate with kernel stats
sar -d -p 1 | grep -E 'sda|dm-0'
Example output analysis:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
sda 0.00 12.00 0.00 450.00 0.00 1800.00 8.00
dm-0 0.00 0.00 0.00 462.00 0.00 1848.00 8.00
Key observations from your data:
- The 559 tps on
/dev/sda
represents device-mapped requests (after merging) - The
dm-0
shows 18,433 tps - these are logical volume requests before merging - Your
ioping
results show physical hardware limitations (92 IOPS)
When sizing storage for applications:
# Calculate required IOPS based on tps:
Effective_IOPS = (Read_IOPS × Read_Ratio) + (Write_IOPS × Write_Ratio)
Write_IOPS = tps × Write_Amplification_Factor
# Typical SATA drive factors:
Sequential_Write_Amplification = 1.0-1.5
Random_Write_Amplification = 2.0-5.0
For your observed 142,600 Blk_wrtn/s (4K blocks ≈ 35,650 IOPS), the physical device would need to handle about 10,000-15,000 actual IOPS after accounting for request merging and write coalescing.