Debugging NTP Synchronization Failures: When ntpdate and ntpd Fail Silently on Linux Servers


2 views

Recently encountered a puzzling scenario where both ntpd and ntpdate failed to synchronize time on a CentOS 6.3 x64 server, despite all connectivity tests passing normally. Here's what I found:

# Basic connectivity test
ntpq -pn pool.ntp.org
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+208.97.140.69    .GPS.            1 u   12   64  377   75.927   -2.123   1.234

The system exhibited several conflicting behaviors:

  • Manual time setting worked: date -s "30 DEC 2012 02:30:00"
  • Debug mode showed correct offsets: ntpdate -d pool.ntp.org
  • Yet normal operation failed with "no server suitable"

After extensive testing, I identified several potential culprits:

# Check current time sources
chronyc sources -v
# Verify NTP process permissions
ls -la /var/lib/ntp/drift

The actual issue turned out to be the system clock being too far out of sync (>1000 seconds). NTP has built-in protection against large jumps:

# Force immediate sync (if offset is < 1000s)
ntpdate -b pool.ntp.org

# For larger offsets, first set time manually:
date -s "2023-01-01 12:00:00"
service ntpd restart

For production systems, I recommend:

# /etc/ntp.conf configuration:
tinker panic 0
server 0.pool.ntp.org iburst
server 1.pool.ntp.org iburst
server 2.pool.ntp.org iburst
server 3.pool.ntp.org iburst

# Then restart the service
service ntpd restart

For modern systems, consider using chrony instead:

yum install chrony
systemctl enable chronyd
systemctl start chronyd
chronyc makestep

I recently encountered a perplexing issue where both ntpd and ntpdate appeared to function normally in debug mode but failed to actually synchronize the system clock. Here's my deep dive into diagnosing and resolving this problem.

Basic diagnostics showed no obvious network or permission issues:

# Manual time setting works
date -s "30 DEC 2012 02:30:00"

# NTP query returns valid responses
ntpq -pn pool.ntp.org

# Debug mode shows time offset calculation
ntpdate -d pool.ntp.org
30 Dec 02:38:56 ntpdate[19267]: step time server 208.97.140.69 offset 228.234554 sec

Despite appearing functional, actual synchronization attempts fail:

# Normal ntpdate command fails
ntpdate pool.ntp.org
30 Dec 02:41:29 ntpdate[19274]: no server suitable for synchronization found

Similarly, ntpd shows all servers stuck in .INIT. state.

1. Verify NTP server accessibility:

# Test UDP connectivity
nc -uzv pool.ntp.org 123

2. Check system clock discipline:

# View current time synchronization status
timedatectl status
# Or on older systems:
hwclock --debug

3. Examine NTP configuration:

# Check configuration file
cat /etc/ntp.conf
# Verify drift file permissions
ls -l /var/lib/ntp/drift

Solution 1: Time Jump Threshold

When the time difference exceeds NTP's default threshold (typically 1000s), it may refuse to sync. Force an update:

ntpdate -b pool.ntp.org

Solution 2: NTPD Configuration

Add these to /etc/ntp.conf:

tinker panic 0
tos maxdist 30

Solution 3: Alternative Time Sync

As a temporary measure, use chrony:

yum install chrony
systemctl enable chronyd
systemctl start chronyd

1. Check system clock source:

# For virtual machines
cat /sys/devices/system/clocksource/clocksource0/current_clocksource

# For physical servers
dmidecode -t processor | grep -i clock

2. Verify kernel timekeeping:

# Check for time-related kernel messages
dmesg | grep -i time
dmesg | grep -i clock

In my case, the issue was caused by a combination of factors:

  1. The initial time offset was too large (>15 minutes)
  2. The system was configured with an overly restrictive tos maxdist value
  3. There was interference from a virtualization hypervisor's time sync

The permanent fix involved:

# Step 1: Force initial sync with large offset
ntpdate -b pool.ntp.org

# Step 2: Modify ntp.conf
echo "tinker panic 0" >> /etc/ntp.conf
echo "tos maxdist 30" >> /etc/ntp.conf

# Step 3: Restart NTPD
service ntpd restart