Resolving NFS Client Cache Coherency Issues During Code Deployments (NFSv3)


2 views

During our PHP application deployments, we encountered a critical issue where NFS clients would serve stale content despite updated files on the server. The problem manifests when:

  • Files are read during deployment
  • NFS client caches maintain outdated versions
  • Subsequent reads return cached (dirty) data
  • Web servers return blank pages until cache clearance

NFSv3's attribute caching (acregmin/acregmax) and data caching (acdirmax/acdirmin) settings determine how long clients hold cached data before checking the server for updates. The default values often cause exactly this deployment headache.

While we initially resolved this by manually clearing caches post-deploy, better solutions exist:

# Flush NFS client cache (temporary fix)
sync; echo 3 > /proc/sys/vm/drop_caches
# Or unmount/remount the NFS share
umount /mnt/nfs && mount /mnt/nfs

For our CentOS servers, these /etc/nfsmount.conf adjustments solved 90% of cases:

[NFSMount]
Defaultvers=3
Acregmin=5
Acregmax=15
Acdirmin=5
Acdirmax=15

For deployments, we now combine this with a pre-deployment script:

#!/bin/bash
# Pre-deploy cache warmer
find /var/www/html -type f -exec cat {} > /dev/null \;
# Forces cache population before deployment begins

For particularly problematic environments, consider:

  1. NFSv4 migration - Better cache coherency protocols
  2. Deploy staging - Atomic directory switching via symlinks
  3. Blue-green deploys - Maintain separate deployment directories

This diagnostic script helps identify stale cache issues:

#!/bin/bash
SERVER_FILE=$(stat -c %Y /nfs/server/path/file.php)
CLIENT_FILE=$(stat -c %Y /mnt/nfs/file.php)
if [ $SERVER_FILE -gt $CLIENT_FILE ]; then
  echo "WARNING: Cache incoherency detected"
  exit 1
fi

During our recent deployment cycles, we encountered a particularly nasty issue where NFS v3 clients would serve stale content even after file updates. The symptoms manifested as blank pages being served until we manually cleared the NFS cache. Here's what we learned about this caching behavior and how we solved it permanently.

The root cause lies in NFS v3's weak cache consistency model. When our deployment process updated files while web servers were actively reading them, the clients maintained dirty cache entries without properly invalidating them. Here's a sequence that illustrates the problem:

1. Web server reads file.php (cached locally)
2. Deployment updates file.php
3. Subsequent requests still serve old cached version
4. Blank pages appear due to version mismatches

After extensive testing, we found several effective approaches:

1. Immediate Cache Invalidation

Add this to your deployment script to force cache updates:

# For Linux clients
echo 3 > /proc/sys/vm/drop_caches

# Alternative for mounted filesystems
sync; sysctl vm.drop_caches=3

2. NFS Mount Options

Modify your /etc/fstab with these recommended options:

server:/export /mnt nfs 
    rw,
    noac,
    lookupcache=none,
    hard,
    intr,
    timeo=600,
    retrans=2,
    vers=3 0 0

3. Application-Level Workaround

For PHP applications, add this to your deployment process:

# Clear opcache after deployment
php -r 'opcache_reset();'

# For Laravel specifically
php artisan optimize:clear

When all else fails, this sequence has never let us down:

# Unmount and remount the filesystem
umount /mnt/nfs
mount -a

# Clear all possible caches
sync; echo 3 > /proc/sys/vm/drop_caches
service nscd restart

To avoid this issue entirely:

  • Schedule deployments during low-traffic periods
  • Implement blue-green deployment patterns
  • Consider moving to NFS v4.1+ for better cache coherency
  • Use distributed cache invalidation systems like Varnish

While NFS v3's caching behavior can be frustrating, understanding its limitations allows us to work around them effectively. The solutions presented here have proven reliable across hundreds of deployments in our environment.