After an unexpected shutdown sequence (hosts after VMs due to HVAC failure), one specific VMFS volume refuses to mount while others on the same Synology NAS function normally. The smoking gun appears in ESXi logs:

2023-11-15T08:22:47.123Z error hostd[2100012] [Originator@6876 sub=Libs opID=ff38-a12] ATS-ONLY VMFS volume 'Synology-Email server' not mounted. Host does not support ATS or ATS initialization failed

The root cause lies in VMware's Atomic Test & Set (ATS) mechanism - a locking protocol for VMFS volumes. Here's what likely happened:

1. This particular LUN was formatted with ATS-only mode (common for newer VMFS versions)
2. During unclean shutdown, the ATS signature became corrupted
3. ESXi 6.0 has known issues with ATS recovery

First, verify the LUN's ATS status via SSH:

esxcli storage core device list -d naa.xxxxxxxxxxxxxxx | grep ATS

If output shows "ATS Status: Not supported", we need to force a rescan with ATS disabled:

esxcli storage core adapter rescan --all
esxcli system settings advanced set -o /VMFS3/UseATSForHBOnVMFS5 -i 0

For stubborn cases, try mounting via CLI bypassing the ATS check:

vmkfstools -V /vmfs/volumes/Synology-Email\ server
vmkfstools --configVolumeOnly /vmfs/volumes/Synology-Email\ server

To avoid recurrence, consider these Synology iSCSI tweaks:

# Disable immediate allocation on Synology:
iscsi_lun -c modify -l lun_name -o noreserve=1

# ESXi advanced settings for better iSCSI recovery:
esxcli system settings advanced set -o /Disk/QFullSampleSize -i 32
esxcli system settings advanced set -o /Disk/QFullThreshold -i 8

As last resort before rebuilding the datastore:

1. Create new LUN on Synology
2. Use vmkfstools to clone the problematic volume:

vmkfstools -i /vmfs/volumes/bad_volume/vm.vmdk /vmfs/volumes/new_volume/vm.vmdk -d thin

When your ESXi hosts booted up after the power cycle, the system encountered a specific storage-related issue with the email server VM datastore. The key error message indicates:

ATS-ONLY VMFS volume 'Synology-Email server' not mounted. 
Host does not support ATS or ATS initialization has failed

ATS (Atomic Test & Set) is a VMware locking mechanism for VMFS volumes. When a VMFS volume is formatted as "ATS-only," it requires this specific feature for proper operation. ESXi 6.0 has certain limitations with ATS implementation, especially when dealing with certain NAS devices.

Interestingly, your other VMs on the same NAS work fine because:

• They might be using different VMFS versions (3 vs 5/6)
• They might have been formatted without the ATS-only flag
• The LUNs might have different advanced features enabled

1. First, verify the VMFS version of the problematic datastore via SSH:

   esxcli storage vmfs extent list

2. Check if ATS is properly initialized on your host:

   esxcli storage core device vaai status get

3. Attempt a forced rescan of all storage adapters:

   esxcli storage core adapter rescan --all

If the immediate steps don't resolve the issue, consider these approaches:

Option 1: Recreate the VMFS Datastore

# First, safely remove the existing datastore
esxcli storage filesystem unmount -l "Synology-Email server"
esxcli storage filesystem remove -l "Synology-Email server"

# Then recreate it with proper VMFS5/6 formatting
esxcli storage filesystem mklabels -l vmfs -d naa.xxxxxxxx
esxcli storage filesystem vmfs new -d naa.xxxxxxxx -V "Email_DataStore"

Option 2: Upgrade ESXi Hosts
ESXi 6.5+ has better ATS support. Consider upgrading with this process:

esxcli software profile update -p ESXi-6.7.0-xxxxxx-standard \
-d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

To avoid similar issues:

• Standardize on VMFS6 for new datastores
• Maintain consistent power management procedures (consider UPS for NAS)
• Implement proper monitoring for storage health

For Synology NAS devices, ensure these settings:

# In DSM Storage Manager:
1. Enable "Enable multiple sessions" for iSCSI LUN
2. Disable "Enable SSD TRIM" if using SSDs
3. Set "Queue Depth" to 64 or higher
4. Enable "VMware VAAI" support