How to Fix “LV Status: Not Available” After iSCSI Target Reconnection in Linux LVM


2 views

When working with iSCSI storage and LVM, I recently encountered a puzzling situation where logical volumes showed as "NOT available" after reconnecting to the same iSCSI target from a different host. Here's the complete workflow that led to the issue:

# Initial setup on Server A
iscsiadm -m node -T iqn.2004-04.com.qnap:ts-509:iscsi.linux01.ba4731 -p 192.168.0.4 -l
pvcreate /dev/sdb
vgcreate vg00 /dev/sdb
lvcreate -L 17G -n vm vg00
mkfs.ext3 /dev/vg00/vm
iscsiadm -m node -T iqn.2004-04.com.qnap:ts-509:iscsi.linux01.ba4731 -p 192.168.0.4 -u

After connecting from Server B:

iscsiadm -m node -T iqn.2004-04.com.qnap:ts-509:iscsi.linux01.ba4731 -p 192.168.0.4 -l
lvdisplay

The output showed the LV status as "NOT available" and the device node was missing.

The issue occurs because:

  • LVM metadata is stored in two locations: on disk and in memory
  • When disconnecting, the in-memory metadata isn't properly synchronized
  • The new host doesn't automatically reactivate existing LVs

Here's how to properly reactivate the logical volume:

# First, rescan the VG to detect changes
vgscan --mknodes

# Then activate all volumes in the VG
vgchange -ay vg00

# Verify the LV status
lvdisplay /dev/vg00/vm

For persistent configuration across reboots, add this to your init scripts or systemd units:

# Example systemd service snippet
[Unit]
Description=Activate iSCSI LVM volumes
After=iscsi.service

[Service]
Type=oneshot
ExecStart=/sbin/vgchange -ay vg00

[Install]
WantedBy=multi-user.target

If the basic activation doesn't work, try these steps:

# Force a metadata refresh
vgcfgrestore -f /etc/lvm/backup/vg00 vg00

# Check for device filters
cat /etc/lvm/lvm.conf | grep filter

# Temporary bypass of device filters (for testing)
lvdisplay --config 'devices { filter = [ "a|.*|" ] }'

To avoid this situation in the future:

  • Always properly export VGs before disconnecting: vgchange -an vg00
  • Consider using LVM tags for iSCSI volumes
  • Implement proper multipath configuration if using multiple connections

When working with iSCSI storage in Linux environments combined with LVM (Logical Volume Manager), we often encounter a puzzling scenario where previously created logical volumes show "NOT available" status after reconnecting the storage to a different host. Here's what's happening under the hood:

# Typical error output you'll see
linux01:~ # lvdisplay 
  --- Logical volume ---
  LV Name                /dev/vg00/vm
  VG Name                vg00
  LV UUID                NBNRGV-FkSR-ZNZ9-9AVk-chLQ-j5nc-RazeBw
  LV Write Access        read/write
  LV Status              NOT available
  LV Size                17.00 GB

The root cause lies in how LVM maintains device mappings. When you initially create the PV/VG/LV structure, LVM stores device information in:

  • /etc/lvm/backup/ - Configuration backups
  • /etc/lvm/archive/ - Versioned archives
  • Volume Group metadata areas

During iSCSI reconnection, even if the block device appears identical (same WWID), Linux may assign it a different /dev/sdX name, causing LVM's device filter to miss it.

Method 1: Full LVM Rescan

The most reliable approach is to force LVM to rebuild its device cache:

# First ensure the iSCSI device is properly connected
ls -l /dev/disk/by-path/*iscsi*

# Clear LVM cache and rescan
pvscan --cache
vgscan --cache
lvscan --cache

# Activate all volume groups
vgchange -ay

Method 2: Manual Device Filter Adjustment

If rescan doesn't work, we need to explicitly tell LVM about the new device path:

# Locate the actual device path
lsblk -o NAME,FSTYPE,MOUNTPOINT,UUID

# Add to LVM filter in /etc/lvm/lvm.conf
filter = [ "a|/dev/sdb1|", "r|.*|" ]

# Then update initramfs (for persistent changes)
update-initramfs -u

Method 3: Low-Level Reactivation

For stubborn cases, we can directly manipulate the VG metadata:

# Dump current VG configuration
vgcfgbackup vg00

# Edit the backup file to reflect new paths
nano /etc/lvm/backup/vg00

# Restore the configuration
vgcfgrestore -f /etc/lvm/backup/vg00 vg00

In enterprise setups, consider adding these udev rules to /etc/udev/rules.d/99-iscsi-lvm.rules:

# Automatically rescan LVM when iSCSI devices appear
ACTION=="add", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_BUS}=="scsi", ENV{ID_SERIAL}=="*iscsi*", RUN+="/usr/sbin/pvscan --cache"
ACTION=="add", SUBSYSTEM=="block", ENV{DEVTYPE}=="partition", ENV{ID_BUS}=="scsi", ENV{ID_SERIAL}=="*iscsi*", RUN+="/usr/sbin/vgchange -ay"

After applying any solution, verify with:

# Check LV status
lvdisplay /dev/vg00/vm | grep "LV Status"

# Verify filesystem accessibility
file -sL /dev/vg00/vm
fsck.ext3 -n /dev/vg00/vm

Remember that these techniques apply not just to iSCSI, but to any storage subsystem where device paths might change between connections (FC, SAS, NVMe-oF).