When running precise benchmarks on Linux systems, the kernel's performance monitoring subsystem (perf) can introduce unwanted variance. The warning message perf interrupt took too long (2504 > 2500) indicates substantial CPU time being consumed by perf sampling operations.

As discovered through kernel configuration exploration, perf becomes mandatory when:

CONFIG_HAVE_PERF_EVENTS=y
CONFIG_PERF_EVENTS=y

These are automatically enabled on x86/x86_64 architectures, making complete removal impossible without kernel source modification.

1. Throttling Sampling Rate

The most effective approach is minimizing the sampling frequency:

# Set maximum sample rate to absolute minimum
sudo sysctl -w kernel.perf_event_max_sample_rate=1

2. Adjusting CPU Time Allocation

Control perf's CPU budget percentage:

# Set conservative CPU time allowance
sudo sysctl -w kernel.perf_cpu_time_max_percent=1

3. Disabling Per-Event Sampling

Prevent specific event monitoring:

# Disable CPU cycle sampling
echo 0 | sudo tee /sys/kernel/debug/tracing/events/perf/enable

For persistent configuration, add these to kernel command line:

perf_event_max_sample_rate=1 perf_cpu_time_max_percent=1

Check current perf impact with:

# View active perf events
sudo perf stat -a sleep 1

# Monitor NMI interrupts
grep "NMI" /proc/interrupts

While not disabling core perf functionality, prevent auxiliary modules:

# Add to /etc/modprobe.d/blacklist.conf
blacklist perf_pmu
blacklist x86_pmu

While complete perf subsystem removal isn't feasible on standard x86 kernels, these mitigation strategies can reduce its benchmarking impact to negligible levels. The combination of minimal sampling rate and strict CPU time limits provides the most effective solution.

---

When running performance benchmarks on Linux systems, the kernel's performance monitoring subsystem (perf) can introduce unwanted variability. The key warning that appears in dmesg:

[236832.221937] perf interrupt took too long (2504 > 2500), lowering kernel.perf_event_max_sample_rate to 50000

indicates that perf sampling is consuming significant CPU resources, potentially affecting benchmark results.

For x86/x86_64 architectures, perf is a mandatory component due to kernel dependencies:

CONFIG_HAVE_PERF_EVENTS=y
CONFIG_PERF_EVENTS=y

The kernel configuration shows perf is selected by:

selected by: X86 || (MIPS && 64BIT) || PPC || S390 || ARM || ARM64 || SPARC64

While complete disabling isn't feasible, we can minimize perf's impact through several sysctl parameters:

# Set maximum sample rate to minimum allowed value
sudo sysctl -w kernel.perf_event_max_sample_rate=1

# Disable perf CPU time monitoring
sudo sysctl -w kernel.perf_cpu_time_max_percent=0

# Disable perf event paranoia (allows non-root users to disable perf)
sudo sysctl -w kernel.perf_event_paranoid=-1

# Limit perf event mlock pages
sudo sysctl -w kernel.perf_event_mlock_kb=0

For systems with perf built as modules, you can prevent loading:

# Add to /etc/modprobe.d/blacklist.conf
blacklist perf
blacklist perf_pmu
blacklist x86_pmu

Add these parameters to GRUB configuration:

noperf
noperf_event

After applying these changes, monitor:

# Check current perf settings
cat /proc/sys/kernel/perf_event_max_sample_rate
cat /proc/sys/kernel/perf_cpu_time_max_percent

# Monitor perf interrupts
grep perf /proc/interrupts

For reproducible benchmarks, also consider:

# Disable CPU frequency scaling
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Disable address space randomization
echo 0 > /proc/sys/kernel/randomize_va_space