Linux Kernel Log Time Mismatch: Solving dmesg Timestamp vs System Clock Discrepancy

During routine system diagnostics, I encountered a puzzling behavior where timestamps in kernel logs (dmesg) showed significant deviation from the system clock. Here's what my investigation revealed:

# Reproducing the issue
echo "TIMESTAMP_TEST" > /dev/kmsg
dmesg -T | tail -1
[Mon Feb 17 04:57:03 2014] TIMESTAMP_TEST

date
Mon Feb 17 11:45:17 CET 2014

Linux maintains two distinct time references:

System Time: Maintained by the kernel, synchronized via NTP
Hardware Clock: (RTC/BIOS time) - Battery-backed real-time clock
Kernel Log Timestamps: Based on monotonic clock since boot

The discrepancy occurs because:

Kernel logs use the system uptime clock (monotonic) for timestamps
This clock initializes from the hardware RTC at boot
If RTC was incorrect at boot time, subsequent NTP adjustments won't affect existing log entries

Option 1: Persistent Hardware Clock Sync

# Sync system time to hardware clock
hwclock --systohc --utc

# Verify synchronization
hwclock --show
date

Option 2: Kernel Parameter Adjustment

# Add to GRUB configuration:
GRUB_CMDLINE_LINUX="clock=tsc tsc=reliable"
update-grub
reboot

Option 3: Scripted Log Correction (Temporary Fix)

#!/bin/bash
# Calculate time difference between system and boot time
DIFF_SECONDS=$(($(date +%s) - $(cat /proc/uptime | cut -d' ' -f1 | cut -d'.' -f1)))
dmesg | while read -r line; do
    ts=$(echo "$line" | sed -n 's/^$\([^]]*$\].*/\1/p')
    if [ -n "$ts" ]; then
        new_ts=$(date -d "@$(($(date -d "$ts" +%s) + $DIFF_SECONDS))")
        echo "[$new_ts] ${line#*] }"
    else
        echo "$line"
    fi
done

Implement automated RTC synchronization in your provisioning process
Consider using chrony instead of ntpd for better hardware clock handling
Monitor time drift via Nagios/Zabbix with custom checks

For deeper investigation:

# Check current time sources
cat /sys/devices/system/clocksource/clocksource0/current_clocksource

# View available clocksources
cat /sys/devices/system/clocksource/clocksource0/available_clocksource

# Check RTC status
timedatectl show | grep RTCTimeUSec

When debugging Linux systems, accurate timestamps in kernel logs (dmesg output) are crucial for correlating events. However, many administrators encounter situations where the kernel ring buffer timestamps don't match the system time:


# Test case showing the discrepancy
$ echo "DEBUG_TEST" > /dev/kmsg
$ dmesg -T | tail -1
[Mon Feb 17 04:57:03 2014] DEBUG_TEST
$ date
Mon Feb 17 11:45:17 CET 2014

The discrepancy occurs because:

System time (date) uses the Linux kernel's timekeeping (affected by NTP)
Kernel log timestamps use the system uptime counter initialized at boot

The 6-7 hour offset in the example strongly suggests:

BIOS/RTC maintains local time instead of UTC
Linux is configured to expect UTC from hardware
No proper time sync occurred during early boot

Option 1: Configure hardware clock properly


# Check current hardware clock mode
$ timedatectl | grep "RTC time"
# Set hardware clock to UTC
$ timedatectl set-local-rtc 0
# Alternative for older systems:
$ hwclock --systohc --utc

Option 2: Apply boot-time time sync


# For systemd systems:
# Edit /etc/systemd/timesyncd.conf
[Time]
NTP=pool.ntp.org
# Then enable early service
$ systemctl enable systemd-timesyncd.service
$ systemctl start systemd-timesyncd.service

# For non-systemd systems using ntpdate:
$ apt-get install ntpdate
$ echo "ntpdate pool.ntp.org" >> /etc/rc.local

For immediate log analysis:


# Calculate the offset between system time and boot time
$ offset=$(($(date +%s) - $(cat /proc/uptime | cut -d. -f1)))
# Then parse dmesg with adjustment
$ dmesg | while read -r line; do
    ts=$(echo "$line" | sed -n 's/^$*\([0-9.]*$.*/\1/p')
    if [ -n "$ts" ]; then
        adjusted=$(date -d "@$(echo "$offset + $ts" | bc)" +"%F %T")
        echo "$line" | sed "s/^$$*[0-9.]*$$/[${adjusted}]/"
    else
        echo "$line"
    fi
done

After applying fixes:


$ echo "VERIFICATION_TEST" > /dev/kmsg
$ sleep 1
$ current_time=$(date "+%F %H:%M")
$ dmesg -T | grep VERIFICATION_TEST | grep "$current_time" && echo "FIXED" || echo "STILL BROKEN"