When dealing with Pacemaker-managed GlusterFS servers in an HA cluster, we face a chicken-and-egg problem during system boot:
1. GlusterFS requires the cluster to be online
2. Cluster services may depend on mounted GlusterFS volumes
3. System boot attempts mounts before Pacemaker initializes
The key is to create a systemd mount unit with proper dependency chains and retry logic. Here's an improved version:
# /etc/systemd/system/data.mount
[Unit]
Description=GlusterFS Mount for /data
Requires=pacemaker.service
After=pacemaker.service
ConditionPathExists=/var/run/pacemaker.pid
StartLimitIntervalSec=60
StartLimitBurst=5
[Mount]
What=nodea:/gv0
Where=/data
Type=glusterfs
Options=_netdev,retry=5,timeo=300
[Install]
WantedBy=multi-user.target
For more robust handling, implement a custom systemd service that performs health checks:
# /etc/systemd/system/glusterfs-mount-retry.service
[Unit]
Description=GlusterFS Mount Retry Service
After=network.target pacemaker.service
[Service]
Type=oneshot
ExecStartPre=/bin/bash -c 'until ping -c1 nodea &>/dev/null; do sleep 2; done'
ExecStart=/bin/mount /data
ExecStartPost=/bin/bash -c 'if ! mountpoint -q /data; then systemctl restart data.mount; fi'
TimeoutStartSec=600
Restart=on-failure
RestartSec=30
[Install]
WantedBy=multi-user.target
Configure Pacemaker to manage the mount as a cluster resource:
# Add to your existing Pacemaker configuration
pcs resource create gluster-mount Filesystem device="nodea:/gv0" directory="/data" \
fstype="glusterfs" options="_netdev" --group cluster-resources
pcs constraint order start virtual-ip then gluster-mount
pcs constraint colocation add gluster-mount with virtual-ip INFINITY
Create a systemd service to verify mount status during boot:
# /etc/systemd/system/verify-gluster-mount.service
[Unit]
Description=Verify GlusterFS Mount
After=data.mount
[Service]
Type=oneshot
ExecStart=/usr/bin/test -d /data/.glusterfs
ExecStart=/usr/bin/grep -q glusterfs /proc/mounts
[Install]
WantedBy=multi-user.target
For environments where systemd integration proves challenging, consider an init script approach:
#!/bin/bash
# /etc/init.d/gluster-mount
RETVAL=0
PROG="gluster-mount"
GLUSTER_VOL="nodea:/gv0"
MOUNT_POINT="/data"
start() {
if ! grep -q "$MOUNT_POINT" /proc/mounts; then
echo -n $"Mounting $GLUSTER_VOL: "
mount -t glusterfs -o _netdev,retry=10 "$GLUSTER_VOL" "$MOUNT_POINT"
RETVAL=$?
[ $RETVAL -eq 0 ] && touch /var/lock/subsys/$PROG
return $RETVAL
fi
}
case "$1" in
start)
start
;;
*)
echo "Usage: $0 {start}"
exit 1
;;
esac
exit $RETVAL
Then enable it with:
chkconfig --add gluster-mount
chkconfig gluster-mount on
When dealing with GlusterFS mounts on clustered servers managed by Pacemaker in RHEL 7, we face a fundamental boot sequence dependency. The traditional approaches fail because:
- fstab attempts mounting before cluster services initialize
- Systemd mount units may execute before Pacemaker establishes the virtual IP
- Concurrent reboots create circular dependencies between nodes
Here's an improved systemd unit file that properly handles cluster dependencies:
# /etc/systemd/system/data.mount
[Unit]
Description=GlusterFS Mount for Cluster Storage
Requires=pacemaker.service
After=pacemaker.service
Before=network.target
ConditionPathExists=/var/lib/pacemaker/cores/root.uid
[Mount]
What=nodea:/gv0
Where=/data
Type=glusterfs
Options=_netdev,backup-volfile-servers=nodeb
TimeoutSec=300
RetrySec=10
[Install]
WantedBy=multi-user.target
The critical improvements in this approach:
- Cluster-aware condition checking: Using ConditionPathExists to verify Pacemaker initialization
- Proper service ordering: Explicit dependencies on pacemaker.service
- Failover support: backup-volfile-servers option for redundancy
- Retry mechanism: Built-in retry logic with TimeoutSec and RetrySec
To validate your implementation:
# Verify unit dependencies
systemctl show data.mount -p Requires,After
# Test mount sequence
systemctl stop data.mount
systemctl start data.mount
journalctl -u data.mount --since "1 hour ago"
# Simulate cluster failure
pcs cluster stop --all
systemctl restart pacemaker
For maximum reliability, consider managing the mount through Pacemaker itself:
pcs resource create gluster-mount Filesystem \
device="nodea:/gv0" \
directory="/data" \
fstype="glusterfs" \
options="_netdev,backup-volfile-servers=nodeb" \
op monitor interval=60s \
--group cluster-storage
Special scenarios to consider:
- Cluster partition situations (use
pcs property set no-quorum-policy=ignore
) - Network delays (adjust
systemd-networkd-wait-online.service
timeout) - Multi-mount protection (implement flock in init scripts if needed)
- Disable fstab entry (comment it out)
- Enable systemd unit:
systemctl enable data.mount
- Verify boot sequence:
systemd-analyze plot > boot.svg
- Test full cluster reboot sequence
- Implement monitoring for mount health