When analyzing Linux system performance with tools like iostat, %iowait represents the percentage of time the CPU was idle while waiting for I/O operations to complete. Your sample output shows an alarming 49.83% iowait, which immediately signals a serious I/O bottleneck.

Here's how to evaluate iowait percentages:

• 0-5%: Normal range for most systems
• 5-20%: Potentially concerning, warrants investigation
• 20-50%: Significant I/O contention
• 50%+: Critical bottleneck requiring immediate action

Your output reveals several critical issues:

dm-3             78.39% utilization
sdc              74.14% utilization

These devices are operating near their maximum capacity, explaining the high iowait. The await values (15.44ms for sdc) further confirm I/O latency problems.

Here's how to investigate further:

# Monitor specific device I/O
iostat -x 1 sdc dm-3

# Check which processes are causing I/O
iotop -oP

# View disk I/O by process
pidstat -d 1

Based on your output, consider these approaches:

1. Filesystem Optimization: For dm-3 with high writes (21,121.22 wsec/s), consider:

# Example tuning for ext4
mount -o remount,noatime,nodiratime,data=writeback /dev/dm-3

2. I/O Scheduler Adjustment:

# For SSD devices like sdc
echo deadline > /sys/block/sdc/queue/scheduler

3. Application-Level Changes: Implement write batching or caching

For continuous monitoring, use this Prometheus exporter config:

# node_exporter textfile collector script
#!/bin/bash
iostat -x 1 2 | awk '/^dm-3/ {print "node_disk_io_util{dm=\"dm-3\"}", $NF}' > /var/lib/node_exporter/iostat.prom

With sustained iowait above 30%, consider:

• Adding faster storage (NVMe instead of SATA)
• Implementing RAID configurations
• Distributing I/O load across multiple devices

Here's what healthy disk I/O looks like:

Device:    %util await svctm
nvme0n1    12.3  0.45  0.12
sda         8.7  1.22  0.85

Notice the significantly lower await times and utilization percentages compared to your system.

Implement this Python script for historical iowait tracking:

import subprocess
import time
import sqlite3

def log_iowait():
    conn = sqlite3.connect('iowait_monitor.db')
    c = conn.cursor()
    c.execute('''CREATE TABLE IF NOT EXISTS iowait 
                 (timestamp DATETIME, value REAL)''')
    
    while True:
        output = subprocess.check_output(["iostat", "-c", "1", "2"])
        lines = output.decode().split('\n')
        iowait = float(lines[-3].split()[3].replace(',','.'))
        
        c.execute("INSERT INTO iowait VALUES (datetime('now'), ?)", (iowait,))
        conn.commit()
        time.sleep(60)

if __name__ == "__main__":
    log_iowait()

The %iowait metric from iostat shows CPU time spent waiting for I/O operations to complete. Your sample output showing 49.83% iowait is definitely problematic - this indicates nearly half of CPU cycles are being wasted waiting on storage.

# Critical thresholds:
# Below 5%:  Normal operation
# 5-20%:     Monitor closely
# Above 20%: Critical performance issue
# Above 50%: Severe storage bottleneck

Your dm-3 device shows 78.39% utilization with high throughput (39,493 rsec/s). Combined with the 49.83% iowait, this suggests:

• Storage subsystem is overwhelmed by I/O requests
• Possible RAID controller saturation
• Disk queue length (avgqu-sz) of 3.07 indicates requests backing up

Here's how to investigate further using common Linux tools:

# Check which processes are causing I/O
iotop -oPa

# Detailed device metrics (run during peak load)
iostat -xmd 1 5

# Check filesystem mount options
grep -E 'sda|sdb|sdc|sdd' /proc/mounts

For similar cases, we've successfully:

1. Added noatime,nodiratime to mount options
2. Implemented LVM cache for frequently accessed data
3. Upgraded from HDD to SSD/NVMe storage
4. Tuned vm.dirty_ratio and vm.dirty_background_ratio

This bash script captures full I/O context:

#!/bin/bash
TIMESTAMP=$(date +%F_%H-%M)
OUTDIR="/var/log/io_profiling"
mkdir -p $OUTDIR

# Capture full system snapshot
iostat -xmt 1 30 > $OUTDIR/iostat_$TIMESTAMP.log &
pid=$!

# Simultaneous process monitoring
iotop -botqk -d 1 -n 30 > $OUTDIR/iotop_$TIMESTAMP.log

# Wait for completion
wait $pid

# Post-process findings
grep -A1 "avg-cpu" $OUTDIR/iostat_$TIMESTAMP.log | 
  grep -v "\-\-" | 
  awk '{print "IOWait Alert: "$4"%"}'