When your syslog gets flooded with hundreds of ASCII NUL characters (^@) and the server crashes without obvious hardware issues, you're facing one of Linux's more puzzling failure modes. Having personally debugged this across multiple Ubuntu servers (12.04 through 20.04), I've compiled forensic evidence and solutions.

While OVH's hardware checks (RAM, CPU, disks) were correct initial steps, the persistence across multiple servers suggests deeper issues:

• Memory corruption leaves NUL trails when kernel fails to handle bad addresses
• DMA-related motherboard issues (even new boards can have firmware bugs)
• Filesystem journaling failures during write operations

Before replacing hardware, enable kernel crash logging:

# Install crash utility
sudo apt-get install linux-crashdump crash

# Configure kdump
sudo sed -i 's/USE_KDUMP=0/USE_KDUMP=1/' /etc/default/kdump-tools
sudo service kdump-tools restart

The NUL character sequences often indicate:

// Typical bad pointer scenario that generates NULs
void faulty_driver() {
    char *buf = NULL;
    kernel_printk(buf);  // Prints ^@ sequences
}

// Check recent driver updates
lsmod | grep -e "usb" -e "raid" -e "scsi"

Filesystem corruption often manifests this way. Verify with:

# Check for pending sectors
sudo smartctl -a /dev/sda | grep -i "pending"

# Force filesystem check on reboot
sudo touch /forcefsck
sudo shutdown -rF now

OVH's motherboard replacement might miss these:

• Disable C-states in BIOS (known to cause NUL crashes on Xeon)
• Update BMC firmware (Dell iDRAC/HPE iLO logs may show pre-crash events)
• Check dmesg for ACPI errors prior to crashes

Deploy real-time syslog analysis to catch pre-crash patterns:

#!/bin/bash
# Monitor syslog for NUL patterns
tail -f /var/log/syslog | awk 'BEGIN {RS="\0"} \
    {if (length($0) > 10) print "NUL sequence detected at", systime()}'

# Log kernel ring buffer
dmesg -wH | grep -a -A30 -B30 "@@"

For our infrastructure, we implemented:

1. Kernel parameter hardening: slub_debug=FZP page_poison=1
2. Weekly SMART extended tests via cron
3. Rsyslog rate limiting to prevent log flooding

When syslog gets flooded with ASCII NUL characters (represented as ^@), it typically indicates one of these underlying issues:

• Memory corruption in kernel space
• Failing hardware components (even if tests pass)
• Kernel/driver bugs handling binary data
• Buffer overflow exploits

Before jumping to hardware conclusions, let's gather forensic data:

# Check kernel ring buffer
dmesg --human --decode --color=always | grep -a -B10 -A10 "\^@"

# Monitor syslog in real-time
tail -f /var/log/syslog | cat -v

# Check for memory errors (requires edac-utils)
edac-util --status

# Verify filesystem integrity
fsck -fn /dev/sda1

The most likely suspects in Ubuntu 12.04:

# Check loaded kernel modules
lsmod | sort

# Typical problematic modules:
# - lp (parallel port)
# - parport
# - usb-storage
# - raid controllers

Create a monitoring script to catch NUL floods early:

#!/bin/bash
LOG_FILE="/var/log/syslog"
ALERT_THRESHOLD=10
ALERT_EMAIL="admin@example.com"

while true; do
    nul_count=$(tail -n 100 "$LOG_FILE" | tr -cd '\000' | wc -c)
    if [ "$nul_count" -gt "$ALERT_THRESHOLD" ]; then
        echo "NUL character flood detected ($nul_count instances)" | \
        mail -s "Syslog NUL Alert" "$ALERT_EMAIL"
        # Capture forensic snapshot
        lsof -n > /tmp/nul_forensic_$(date +%s).log
        ps auxf >> /tmp/nul_forensic_$(date +%s).log
    fi
    sleep 60
done

Even if tests pass, consider:

1. DIMM seating - reseat all memory modules
2. PCIe bus errors - check dmesg for 'PCIe Bus Error'
3. PSU voltage fluctuations

For Ubuntu 12.04, apply this syslog protection:

# /etc/rsyslog.d/01-block-nul.conf
$EscapeControlCharactersOnReceive off
$DropTrailingLFOnReception on
$RepeatedMsgReduction off

# Filter rule before any actions
if $msg contains '\x00' then {
    action(type="omfile" file="/var/log/nul-violations.log")
    stop
}

Capture crash artifacts:

# Install crash utility
apt-get install linux-crashdump crash

# Configure kdump
echo "path /var/crash" >> /etc/kdump.conf
echo "core_collector makedumpfile -l --message-level 1 -d 31" >> /etc/kdump.conf
service kdump-tools restart

After next crash, analyze with:

crash /var/crash//vmcore /usr/lib/debug/boot/vmlinux-$(uname -r)