Understanding Disk TPS vs IOPS: Benchmarking and Measurement Techniques for Linux Systems


2 views

When analyzing disk performance on Linux systems, two metrics often cause confusion:

  • IOPS (Input/Output Operations Per Second): Measures the number of distinct read/write operations
  • TPS (Transfers Per Second) from sar/iostat: Counts the number of transactions between the device driver and storage
Device:            tps   Blk_read/s   Blk_wrtn/s
sda             559.00         0.00    142600.00
dm-0          18433.00         0.00    147464.00

The high TPS values (especially for dm devices) occur because:

  1. A single logical I/O can generate multiple physical transactions
  2. Device mapper layers (like LVM) split operations further
  3. Kernel block layer merges/splits requests

For accurate IOPS measurement:

# Measure random read IOPS (4KB blocks)
fio --name=random-read --ioengine=libaio --rw=randread \
--bs=4k --direct=1 --size=1G --numjobs=1 \
--runtime=30 --time_based --end_fsync=1

# Measure sequential write throughput
fio --name=seq-write --ioengine=libaio --rw=write \
--bs=128k --direct=1 --size=1G --numjobs=1 \
--runtime=30 --time_based --end_fsync=1
Metric What It Measures Typical Values
tps Transactions per second Depends on I/O pattern
Blk_read/s Blocks read per second 512-byte blocks
Blk_wrtn/s Blocks written per second 512-byte blocks
%iowait CPU waiting for I/O 5-10% is normal

Several factors affect actual disk performance:

  • Queue depth: Higher depths enable more parallel operations
    # Check queue depth
    cat /sys/block/sda/queue/nr_requests
  • Filesystem overhead: EXT4 vs XFS vs Btrfs
  • I/O scheduler: Deadline, CFQ, or NOOP
    # Check current scheduler
    cat /sys/block/sda/queue/scheduler

For production systems:

  1. Use fio for realistic benchmarks
  2. Monitor await and svctm in iostat -x
  3. Consider filesystem journaling impact
  4. Test with production-like workload patterns

When examining disk performance metrics in Linux systems, it's crucial to understand that IOPS (Input/Output Operations Per Second) and tps (Transfers Per Second) represent different concepts despite their apparent similarity.

IOPS measures the raw number of read/write operations the storage device can handle, while tps (as reported by tools like sar and iostat) counts logical I/O requests issued to the device driver:

# Conceptual breakdown:
Physical IOPS (hardware capability) ≤ Device tps (kernel requests) ≤ Filesystem IOPS (VFS layer)

The discrepancy you're observing stems from several architectural factors:

  1. Block Device Layer Abstraction: The kernel merges adjacent requests and splits large ones
  2. Filesystem Overhead: Journaling, metadata operations, and write barriers create additional transactions
  3. Device Mapper Impact: Your dm-* devices show how LVM/RAID layers multiply underlying I/O

For accurate IOPS measurement, combine these tools:

# Raw device testing (bypass filesystem cache)
fio --name=randread --ioengine=libaio --rw=randread --bs=4k \
    --direct=1 --size=1G --numjobs=1 --runtime=60 --time_based

# Correlate with kernel stats
sar -d -p 1 | grep -E 'sda|dm-0'

Example output analysis:

Device:    rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s  avgrq-sz
sda           0.00    12.00    0.00  450.00     0.00  1800.00     8.00
dm-0          0.00     0.00    0.00  462.00     0.00  1848.00     8.00

Key observations from your data:

  • The 559 tps on /dev/sda represents device-mapped requests (after merging)
  • The dm-0 shows 18,433 tps - these are logical volume requests before merging
  • Your ioping results show physical hardware limitations (92 IOPS)

When sizing storage for applications:

# Calculate required IOPS based on tps:
Effective_IOPS = (Read_IOPS × Read_Ratio) + (Write_IOPS × Write_Ratio)
Write_IOPS = tps × Write_Amplification_Factor

# Typical SATA drive factors:
Sequential_Write_Amplification = 1.0-1.5
Random_Write_Amplification = 2.0-5.0

For your observed 142,600 Blk_wrtn/s (4K blocks ≈ 35,650 IOPS), the physical device would need to handle about 10,000-15,000 actual IOPS after accounting for request merging and write coalescing.