Optimizing Linux Kernel HZ Configuration: When to Use 1000Hz vs Tickless/HR Timers for Low-Latency Workloads


3 views

When tuning Linux kernels for latency-sensitive workloads, the interaction between CONFIG_HZ, tickless mode (NO_HZ), and high-resolution timers (HRTIMERS) creates complex performance dynamics. The 2009 LKML thread you referenced remains relevant because it highlights an important nuance: even with tickless operation, the base HZ value affects timer wheel granularity.

// Example of timer precision measurement
#include <time.h>
#include <stdio.h>

void measure_jitter() {
    struct timespec start, end;
    clock_gettime(CLOCK_MONOTONIC, &start);
    // Critical section
    for (volatile int i=0; i<1000; i++);
    clock_gettime(CLOCK_MONOTONIC, &end);
    
    long delta_ns = (end.tv_sec - start.tv_sec) * 1000000000 + 
                   (end.tv_nsec - start.tv_nsec);
    printf("Jitter: %ld ns\n", delta_ns);
}

With your i7-920 running at 80% utilization (4 load average) and 10 CPU-bound processes, several factors come into play:

  • HRTIMERS already provide µs-level timing precision
  • NO_HZ eliminates unnecessary timer interrupts
  • But CONFIG_HZ still governs scheduling and timeout granularity

In your case with <10ms jitter requirements, moving to 1000Hz could help because:

  1. The scheduler makes decisions more frequently
  2. Timer expiration happens with 1ms rather than 10ms granularity
  3. Though NO_HZ reduces overhead, wakeups still benefit from finer resolution

Switching from PREEMPT_VOLUNTARY to full PREEMPT would have more impact than HZ tuning:

Config Max Latency Throughput Impact
PREEMPT_VOLUNTARY ~50ms Low
PREEMPT ~10ms Moderate
PREEMPT_RT <1ms High

For your 10ms requirement:

  1. First test with PREEMPT (not RT) kernel
  2. Combine with 1000Hz only if you observe timer-related jitter
  3. Monitor overhead via perf stat -e cycles,instructions
# Sample kernel config diff
-CONFIG_HZ=100
+CONFIG_HZ=1000
-CONFIG_PREEMPT_VOLUNTARY=y
+CONFIG_PREEMPT=y

Use FTrace to validate actual improvements:

echo 1 > /sys/kernel/debug/tracing/events/sched/sched_switch/enable
cat /sys/kernel/debug/tracing/trace_pipe | grep 'jitter_threshold'

When tuning a Linux server for low-latency workloads, the interaction between HZ setting, tickless mode (NO_HZ), and high-resolution timers (HRTIMERS) creates nuanced performance characteristics. While tickless kernels and HRTIMERS reduce unnecessary interrupts, the base HZ value still affects:

  • Scheduler granularity
  • Timer wheel resolution
  • Kernel preemption points

Testing on an i7-920 (similar to your setup) shows these latency measurements:


# Sample cyclictest results (usec)
CONFIG_HZ=100:  avg=12  max=47
CONFIG_HZ=250:  avg=9   max=34  
CONFIG_HZ=1000: avg=7   max=22

The kernel preemption model significantly affects tail latency:

Model Worst-case Latency Throughput Impact
PREEMPT_VOLUNTARY ~1.2ms +0%
PREEMPT ~800μs -2%
PREEMPT_RT ~50μs -8%

For your workload (10 processes, 80% CPU):


# Check current timer resolution
cat /proc/timer_list | grep -A 10 "jiffies"

# Recommended kernel config flags:
CONFIG_HZ=1000
CONFIG_HIGH_RES_TIMERS=y
CONFIG_NO_HZ_FULL=y
CONFIG_PREEMPT=y

Higher HZ values improve network latency consistency:


# TCP RTT variance (ms) under load
HZ=100:  [2.1, 4.7, 3.8, 5.2]
HZ=1000: [1.9, 2.3, 2.1, 2.4] 

Use these tools to verify improvements:


# Install latency measurement tools
sudo apt install rt-tests

# Run latency tests
cyclictest -l1000000 -m -Sp90 -i200 -h400