When working with iSCSI storage and LVM, I recently encountered a puzzling situation where logical volumes showed as "NOT available" after reconnecting to the same iSCSI target from a different host. Here's the complete workflow that led to the issue:
# Initial setup on Server A
iscsiadm -m node -T iqn.2004-04.com.qnap:ts-509:iscsi.linux01.ba4731 -p 192.168.0.4 -l
pvcreate /dev/sdb
vgcreate vg00 /dev/sdb
lvcreate -L 17G -n vm vg00
mkfs.ext3 /dev/vg00/vm
iscsiadm -m node -T iqn.2004-04.com.qnap:ts-509:iscsi.linux01.ba4731 -p 192.168.0.4 -u
After connecting from Server B:
iscsiadm -m node -T iqn.2004-04.com.qnap:ts-509:iscsi.linux01.ba4731 -p 192.168.0.4 -l
lvdisplay
The output showed the LV status as "NOT available" and the device node was missing.
The issue occurs because:
- LVM metadata is stored in two locations: on disk and in memory
- When disconnecting, the in-memory metadata isn't properly synchronized
- The new host doesn't automatically reactivate existing LVs
Here's how to properly reactivate the logical volume:
# First, rescan the VG to detect changes
vgscan --mknodes
# Then activate all volumes in the VG
vgchange -ay vg00
# Verify the LV status
lvdisplay /dev/vg00/vm
For persistent configuration across reboots, add this to your init scripts or systemd units:
# Example systemd service snippet
[Unit]
Description=Activate iSCSI LVM volumes
After=iscsi.service
[Service]
Type=oneshot
ExecStart=/sbin/vgchange -ay vg00
[Install]
WantedBy=multi-user.target
If the basic activation doesn't work, try these steps:
# Force a metadata refresh
vgcfgrestore -f /etc/lvm/backup/vg00 vg00
# Check for device filters
cat /etc/lvm/lvm.conf | grep filter
# Temporary bypass of device filters (for testing)
lvdisplay --config 'devices { filter = [ "a|.*|" ] }'
To avoid this situation in the future:
- Always properly export VGs before disconnecting:
vgchange -an vg00
- Consider using LVM tags for iSCSI volumes
- Implement proper multipath configuration if using multiple connections
When working with iSCSI storage in Linux environments combined with LVM (Logical Volume Manager), we often encounter a puzzling scenario where previously created logical volumes show "NOT available" status after reconnecting the storage to a different host. Here's what's happening under the hood:
# Typical error output you'll see
linux01:~ # lvdisplay
--- Logical volume ---
LV Name /dev/vg00/vm
VG Name vg00
LV UUID NBNRGV-FkSR-ZNZ9-9AVk-chLQ-j5nc-RazeBw
LV Write Access read/write
LV Status NOT available
LV Size 17.00 GB
The root cause lies in how LVM maintains device mappings. When you initially create the PV/VG/LV structure, LVM stores device information in:
/etc/lvm/backup/
- Configuration backups/etc/lvm/archive/
- Versioned archives- Volume Group metadata areas
During iSCSI reconnection, even if the block device appears identical (same WWID), Linux may assign it a different /dev/sdX
name, causing LVM's device filter to miss it.
Method 1: Full LVM Rescan
The most reliable approach is to force LVM to rebuild its device cache:
# First ensure the iSCSI device is properly connected
ls -l /dev/disk/by-path/*iscsi*
# Clear LVM cache and rescan
pvscan --cache
vgscan --cache
lvscan --cache
# Activate all volume groups
vgchange -ay
Method 2: Manual Device Filter Adjustment
If rescan doesn't work, we need to explicitly tell LVM about the new device path:
# Locate the actual device path
lsblk -o NAME,FSTYPE,MOUNTPOINT,UUID
# Add to LVM filter in /etc/lvm/lvm.conf
filter = [ "a|/dev/sdb1|", "r|.*|" ]
# Then update initramfs (for persistent changes)
update-initramfs -u
Method 3: Low-Level Reactivation
For stubborn cases, we can directly manipulate the VG metadata:
# Dump current VG configuration
vgcfgbackup vg00
# Edit the backup file to reflect new paths
nano /etc/lvm/backup/vg00
# Restore the configuration
vgcfgrestore -f /etc/lvm/backup/vg00 vg00
In enterprise setups, consider adding these udev rules to /etc/udev/rules.d/99-iscsi-lvm.rules
:
# Automatically rescan LVM when iSCSI devices appear
ACTION=="add", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_BUS}=="scsi", ENV{ID_SERIAL}=="*iscsi*", RUN+="/usr/sbin/pvscan --cache"
ACTION=="add", SUBSYSTEM=="block", ENV{DEVTYPE}=="partition", ENV{ID_BUS}=="scsi", ENV{ID_SERIAL}=="*iscsi*", RUN+="/usr/sbin/vgchange -ay"
After applying any solution, verify with:
# Check LV status
lvdisplay /dev/vg00/vm | grep "LV Status"
# Verify filesystem accessibility
file -sL /dev/vg00/vm
fsck.ext3 -n /dev/vg00/vm
Remember that these techniques apply not just to iSCSI, but to any storage subsystem where device paths might change between connections (FC, SAS, NVMe-oF).