Real-time Linux Kernel Log Monitoring: Techniques for Live dmesg and syslog Output


2 views

When debugging kernel modules or low-level system issues, the ability to monitor logs in real-time is crucial. Linux provides several mechanisms for live log monitoring:

# Traditional syslog location (may vary by distro)
tail -f /var/log/syslog

# Kernel ring buffer access
dmesg -w

# Systemd journal (modern distributions)
journalctl -f -k

The most direct method for kernel log monitoring is using dmesg with the follow mode:

sudo dmesg -w

This continuously displays new kernel messages as they're generated. The -w flag was introduced in util-linux 2.90+ and works by:

  • Reading from /proc/kmsg
  • Maintaining persistent connection
  • Preserving timestamp formatting

For systems using systemd (most modern distributions), journalctl offers powerful filtering:

# Follow kernel logs specifically
journalctl -f -k

# Filter by facility
journalctl -f --facility=kern

# Combine with priority filtering
journalctl -k -p err -f

For maximum control, you can read directly from the kernel message buffer:

# Simple cat approach (requires root)
sudo cat /proc/kmsg

# Python reader example
import time
with open('/proc/kmsg', 'r') as kmsg:
    while True:
        line = kmsg.readline()
        if line:
            print(line.strip())
        time.sleep(0.1)

When dealing with specific kernel modules or subsystems:

# Watch only your module's messages
dmesg -w | grep "your_module_name"

# Combined timestamp and priority filtering
journalctl -k -f --since "10 minutes ago" --grep="sensor"

For high-frequency logging (like sensor data):

  • Use --follow instead of -f in journalctl for better performance
  • Consider rate limiting in your kernel module with printk_ratelimit()
  • Monitor system load with vmstat 1 during logging

When debugging kernel modules or low-level system components, the ability to monitor logs in real-time is crucial. The Linux kernel outputs messages through the kernel ring buffer, which can be accessed via several methods.

The most straightforward approach uses dmesg with its follow mode:

sudo dmesg --follow

This will:

  • Display existing kernel messages
  • Continue running and print new messages as they arrive
  • Preserve timestamps with -T flag for human-readable format

For systems using systemd, journalctl provides enhanced filtering:

sudo journalctl -f -k

Where:

  • -f follows new entries (like tail -f)
  • -k shows only kernel messages

For raw access to /proc/kmsg (requires root):

sudo tail -f /proc/kmsg

Note this:

  • Provides unfiltered kernel messages
  • Can only be read by one process at a time
  • Shows messages at KERN_EMERG through KERN_DEBUG levels

Combine with grep for specific module debugging:

dmesg --follow | grep "your_module_name"

Or with priority filtering:

dmesg --follow --level=err,warn

Ensure your syslog configuration preserves kernel messages:

# In /etc/rsyslog.conf or similar
kern.* /var/log/kern.log

Example of logging from your kernel module:

#include <linux/kernel.h>

printk(KERN_INFO "Sensor reading: %d\n", sensor_value);