When working with Linux software RAID (mdadm), few things are more disconcerting than finding your carefully configured /dev/md0 has suddenly transformed into /dev/md127. This behavior typically occurs during system events like log rotation or service restarts, as evidenced by the syslog entries:

DeviceDisappeared /dev/md0
NewArray /dev/md127

The key to understanding this behavior lies in how mdadm handles array assembly during system events. When mdadm --assemble --scan runs (often triggered by cron jobs or system services), it follows specific rules:

1. First checks /etc/mdadm/mdadm.conf for array definitions
2. Then scans devices for superblock information
3. Uses metadata version to determine naming

The shift to md127 typically occurs when the array is being assembled from device metadata rather than the config file. Modern mdadm versions (particularly with metadata 1.2) may exhibit this behavior when:

• The config file isn't properly updated after array creation
• There's a mismatch between the config and actual array metadata
• Automated assembly processes override manual configurations

To diagnose your specific situation, run these commands:

# Check current array status
cat /proc/mdstat

# Verify metadata version on devices
mdadm --examine /dev/sd[bcdefghijk]1 | grep "Version"

# Check config file contents
cat /etc/mdadm/mdadm.conf

Here's how to permanently resolve the naming issue:

# First, stop the array
mdadm --stop /dev/md127

# Update the array definition (assuming metadata 1.2)
mdadm --assemble --verbose /dev/md0 /dev/sd[bcdefghijk]1 --update=super-minor

# Update the config file
mdadm --detail --scan >> /etc/mdadm/mdadm.conf

# Update initramfs (for Ubuntu/Debian)
update-initramfs -u

To ensure your array maintains its proper name across reboots and system events:

# Add this to your mdadm.conf
ARRAY /dev/md0 metadata=1.2 name=yourhostname:0 UUID=your-array-uuid

# Verify with:
mdadm --examine --scan

# Set proper permissions on config file
chmod 644 /etc/mdadm/mdadm.conf

For systems where the above doesn't work consistently, create a udev rule:

# /etc/udev/rules.d/65-md-raid.rules
SUBSYSTEM=="block", ENV{ID_FS_TYPE}=="linux_raid_member", \
    ENV{ID_FS_UUID}=="your-array-uuid", \
    SYMLINK+="md/%k"

# Then reload udev rules
udevadm control --reload-rules

After implementing changes, verify with:

# Check array status
mdadm --detail /dev/md0

# Force a re-scan
mdadm --assemble --scan --verbose

# Simulate a service restart
service mdadm restart

Remember to test after reboots to ensure persistence.

I recently encountered a strange behavior on an Ubuntu 12.04 LTS server where my perfectly configured RAID10 array suddenly appeared as /dev/md127 instead of /dev/md0, despite all configurations pointing to md0. The syslog showed these simultaneous events:

DeviceDisappeared /dev/md0
NewArray /dev/md127

After digging through kernel documentation and mdadm source code, I found this occurs when:

• The array isn't properly defined in initramfs
• There's a mismatch between /etc/mdadm/mdadm.conf and the actual array metadata
• The system falls back to auto-assembly with the "magic number" 127

The timing with logrotate isn't coincidental. When logrotate restarts syslog, it can trigger udev events that cause the array to reassemble. Check your logrotate config for any services that might affect storage:

/var/log/syslog {
    rotate 7
    daily
    missingok
    notifempty
    delaycompress
    compress
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

Here's how I permanently resolved the issue:

# Stop the array first
sudo mdadm --stop /dev/md127

# Examine superblock details
sudo mdadm --examine /dev/sd[b-k]1 | grep -E "UUID|Name"

# Reassemble with explicit name
sudo mdadm --assemble --name=0 /dev/md0 /dev/sd[b-k]1 --verbose

# Update mdadm.conf
sudo mdadm --detail --scan > /etc/mdadm/mdadm.conf

# Rebuild initramfs
sudo update-initramfs -u

# Verify persistent naming
sudo mdadm --detail /dev/md0 | grep "Name"

Add these checks to your monitoring system:

#!/bin/bash
# RAID array name watchdog
EXPECTED_NAME="md0"
CURRENT_NAME=$(cat /proc/mdstat | awk '/^md/ {print $1; exit}')

if [ "$CURRENT_NAME" != "$EXPECTED_NAME" ]; then
    logger -t raidwatch "RAID array name changed from $EXPECTED_NAME to $CURRENT_NAME"
    # Add recovery logic here
fi

For production systems, consider these kernel parameters in /etc/default/grub:

GRUB_CMDLINE_LINUX="... md=0,/dev/sdb1,/dev/sdc1,/dev/sdd1,..."

This ensures the kernel knows exactly how to assemble your array during early boot.