Automounting GlusterFS in Pacemaker HA Cluster: Solving Boot-Time Mount Dependencies on RHEL 7


15 views

When dealing with Pacemaker-managed GlusterFS servers in an HA cluster, we face a chicken-and-egg problem during system boot:

1. GlusterFS requires the cluster to be online
2. Cluster services may depend on mounted GlusterFS volumes
3. System boot attempts mounts before Pacemaker initializes

The key is to create a systemd mount unit with proper dependency chains and retry logic. Here's an improved version:

# /etc/systemd/system/data.mount
[Unit]
Description=GlusterFS Mount for /data
Requires=pacemaker.service
After=pacemaker.service
ConditionPathExists=/var/run/pacemaker.pid
StartLimitIntervalSec=60
StartLimitBurst=5

[Mount]
What=nodea:/gv0
Where=/data
Type=glusterfs
Options=_netdev,retry=5,timeo=300

[Install]
WantedBy=multi-user.target

For more robust handling, implement a custom systemd service that performs health checks:

# /etc/systemd/system/glusterfs-mount-retry.service
[Unit]
Description=GlusterFS Mount Retry Service
After=network.target pacemaker.service

[Service]
Type=oneshot
ExecStartPre=/bin/bash -c 'until ping -c1 nodea &>/dev/null; do sleep 2; done'
ExecStart=/bin/mount /data
ExecStartPost=/bin/bash -c 'if ! mountpoint -q /data; then systemctl restart data.mount; fi'
TimeoutStartSec=600
Restart=on-failure
RestartSec=30

[Install]
WantedBy=multi-user.target

Configure Pacemaker to manage the mount as a cluster resource:

# Add to your existing Pacemaker configuration
pcs resource create gluster-mount Filesystem device="nodea:/gv0" directory="/data" \
    fstype="glusterfs" options="_netdev" --group cluster-resources
pcs constraint order start virtual-ip then gluster-mount
pcs constraint colocation add gluster-mount with virtual-ip INFINITY

Create a systemd service to verify mount status during boot:

# /etc/systemd/system/verify-gluster-mount.service
[Unit]
Description=Verify GlusterFS Mount
After=data.mount

[Service]
Type=oneshot
ExecStart=/usr/bin/test -d /data/.glusterfs
ExecStart=/usr/bin/grep -q glusterfs /proc/mounts

[Install]
WantedBy=multi-user.target

For environments where systemd integration proves challenging, consider an init script approach:

#!/bin/bash
# /etc/init.d/gluster-mount

RETVAL=0
PROG="gluster-mount"
GLUSTER_VOL="nodea:/gv0"
MOUNT_POINT="/data"

start() {
    if ! grep -q "$MOUNT_POINT" /proc/mounts; then
        echo -n $"Mounting $GLUSTER_VOL: "
        mount -t glusterfs -o _netdev,retry=10 "$GLUSTER_VOL" "$MOUNT_POINT"
        RETVAL=$?
        [ $RETVAL -eq 0 ] && touch /var/lock/subsys/$PROG
        return $RETVAL
    fi
}

case "$1" in
    start)
        start
        ;;
    *)
        echo "Usage: $0 {start}"
        exit 1
        ;;
esac
exit $RETVAL

Then enable it with:

chkconfig --add gluster-mount
chkconfig gluster-mount on

When dealing with GlusterFS mounts on clustered servers managed by Pacemaker in RHEL 7, we face a fundamental boot sequence dependency. The traditional approaches fail because:

  • fstab attempts mounting before cluster services initialize
  • Systemd mount units may execute before Pacemaker establishes the virtual IP
  • Concurrent reboots create circular dependencies between nodes

Here's an improved systemd unit file that properly handles cluster dependencies:

# /etc/systemd/system/data.mount
[Unit]
Description=GlusterFS Mount for Cluster Storage
Requires=pacemaker.service
After=pacemaker.service
Before=network.target
ConditionPathExists=/var/lib/pacemaker/cores/root.uid

[Mount]
What=nodea:/gv0
Where=/data
Type=glusterfs
Options=_netdev,backup-volfile-servers=nodeb
TimeoutSec=300
RetrySec=10

[Install]
WantedBy=multi-user.target

The critical improvements in this approach:

  • Cluster-aware condition checking: Using ConditionPathExists to verify Pacemaker initialization
  • Proper service ordering: Explicit dependencies on pacemaker.service
  • Failover support: backup-volfile-servers option for redundancy
  • Retry mechanism: Built-in retry logic with TimeoutSec and RetrySec

To validate your implementation:

# Verify unit dependencies
systemctl show data.mount -p Requires,After

# Test mount sequence
systemctl stop data.mount
systemctl start data.mount
journalctl -u data.mount --since "1 hour ago"

# Simulate cluster failure
pcs cluster stop --all
systemctl restart pacemaker

For maximum reliability, consider managing the mount through Pacemaker itself:

pcs resource create gluster-mount Filesystem \
    device="nodea:/gv0" \
    directory="/data" \
    fstype="glusterfs" \
    options="_netdev,backup-volfile-servers=nodeb" \
    op monitor interval=60s \
    --group cluster-storage

Special scenarios to consider:

  • Cluster partition situations (use pcs property set no-quorum-policy=ignore)
  • Network delays (adjust systemd-networkd-wait-online.service timeout)
  • Multi-mount protection (implement flock in init scripts if needed)
  1. Disable fstab entry (comment it out)
  2. Enable systemd unit: systemctl enable data.mount
  3. Verify boot sequence: systemd-analyze plot > boot.svg
  4. Test full cluster reboot sequence
  5. Implement monitoring for mount health