When your Linux system shows high load averages despite low CPU utilization and minimal disk I/O, you're likely dealing with processes stuck in the run queue. The run queue contains all processes that are ready to execute but waiting for CPU time.

Here are the most effective commands to examine run queue contents:

# Show running processes and their state
ps -eo pid,state,cmd | grep -v "\["

# Alternative method showing process states
ps -e -o pid,state,cmd | awk '$2 ~ /R|D/ {print}'

For advanced analysis, SystemTap provides unparalleled visibility:

# SystemTap script to monitor runqueue
probe scheduler.runqueue {
    printf("runqueue length: %d\n", $rq->nr_running)
    foreach(task = $rq->task_list) {
        printf("  task: %s (%d)\n", task_execname(task), task_pid(task))
    }
}

Key states to watch for in ps output:

• R: Runnable (in run queue)
• D: Uninterruptible sleep (often I/O)
• S: Interruptible sleep

This Bash script logs processes when run queue exceeds threshold:

#!/bin/bash
THRESHOLD=5
while true; do
    LOAD=$(awk '{print $1}' /proc/loadavg)
    if (( $(echo "$LOAD > $THRESHOLD" | bc -l) )); then
        echo "High load detected: $LOAD"
        ps -eo pid,state,cmd --sort=-state | head -n 10
        echo "--------------------"
    fi
    sleep 5
done

For kernel developers, examining /proc/sched_debug reveals detailed scheduler information:

cat /proc/sched_debug | grep -A10 "runnable"

Remember that frequent polling of run queue status can itself impact performance. For production systems, consider:

• Setting appropriate sampling intervals
• Using kernel tracepoints instead of process polling
• Implementing eBPF-based monitoring for lower overhead

You're seeing high server load averages (reported by tools like sar or top) while your CPU appears largely idle and disk I/O remains low. This classic symptom indicates processes are waiting in the run queue - but what exactly is queued up?

# Basic process states with ps
ps -eo stat,pid,user,comm | grep '^D'
  
# Detailed view with perf
sudo perf sched latency
  
# Kernel thread states
watch -n 1 "grep 'procs_blocked\|procs_running' /proc/stat"

For precise run queue analysis, Linux's ftrace provides unparalleled visibility:

# Set up ftrace
echo function_graph > /sys/kernel/debug/tracing/current_tracer
echo sched_switch >> /sys/kernel/debug/tracing/set_event
echo 1 > /sys/kernel/debug/tracing/tracing_on
  
# Capture run queue activity for 5 seconds
sleep 5
echo 0 > /sys/kernel/debug/tracing/tracing_on
  
# View results
cat /sys/kernel/debug/tracing/trace_pipe | grep "=>"

When examining run queue contents, watch for these patterns:

• D-state processes: Uninterruptible sleep (often I/O wait)
• SIGSTOPped processes: Temporarily removed from scheduling
• Kernel worker threads: Often handling deferred work

This Bash script continuously monitors run queue depth and contents:

#!/bin/bash
while true; do
  RUNQUEUE=$(vmstat 1 2 | tail -1 | awk '{print $1}')
  BLOCKED=$(grep procs_blocked /proc/stat | awk '{print $2}')
  echo "[$(date)] Run queue: $RUNQUEUE | Blocked: $BLOCKED"
  ps -eo stat,pid,user,comm | grep -E '^D|^T'
  sleep 5
done

For modern kernels, BPF tools like runqlat provide precise metrics:

# Install bcc tools
sudo apt install bpfcc-tools
  
# Monitor run queue latency
sudo runqlat-bpfcc
  
# Show what's actually in the queue
sudo runqlen-bpfcc