NFS Cache Invalidation Issue: Client Fails to Detect Server-Side File Modifications Without Directory Listing


2 views

In my NFSv4 setup, I'm observing an unusual caching behavior where client machines don't automatically detect file content changes made directly on the server until a directory listing (ls) command is executed on the client side. This creates a problematic inconsistency in distributed environments.

# Server-side modification (example)
echo "new content" > /nfs_share/testfile.txt

# Client-side observation (initially shows old content)
cat /mnt/nfs/testfile.txt  # Returns previous content
ls /mnt/nfs/               # Triggers cache invalidation
cat /mnt/nfs/testfile.txt  # Now returns updated content

This appears to be related to NFS attribute caching mechanisms. While NFS clients typically cache file attributes (size, modification time) for performance, they should invalidate these caches when changes occur. The lookupcache parameter (defaulting to 'all') might be contributing to this behavior.

Option 1: Mount Parameter Adjustment

# Add these to client mount options in /etc/fstab:
server:/export /mnt/nfs nfs rw,noac,lookupcache=none 0 0

Option 2: Forced Cache Invalidation

# Programmatic solution (Python example):
import os
import time

def refresh_nfs_file(path):
    """Force NFS cache invalidation"""
    os.stat(path)  # Force attribute check
    parent_dir = os.path.dirname(path)
    os.listdir(parent_dir)  # Mimic 'ls' effect

For production systems where complete cache disabling isn't desirable, consider these fine-grained controls:

# Example of tuned parameters:
mount -t nfs -o acregmin=0,acregmax=30,acdirmin=0,acdirmax=30 server:/export /mnt/nfs

This caching behavior can cause significant issues in scenarios like:

  • CI/CD pipelines where build servers modify artifacts
  • Database systems using NFS for shared storage
  • Distributed logging systems

In my NFSv4 setup with one server and multiple clients, I noticed an interesting caching behavior:

  • Client-to-server modifications propagate immediately
  • Server-side modifications require client-side ls to become visible

The root cause lies in how NFS handles file attributes and caching. The default NFS caching parameters:

acregmin=3  acregmax=60
acdirmin=30 acdirmax=60
lookupcache=all

These settings mean:

  • File attributes cached for 3-60 seconds
  • Directory attributes cached for 30-60 seconds
  • Positive lookups are cached

To immediately see server-side changes, try these approaches:

Option 1: Manual Cache Busting

# Force directory cache update
ls -l /mnt/nfs/directory

# Alternative for single file
cat /mnt/nfs/file > /dev/null

Option 2: Mount Options

Modify your /etc/fstab with stricter caching controls:

server:/export /mnt/nfs nfs 
    rw,hard,intr,
    noac,lookupcache=none 0 0

Key parameters:

  • noac: Disables attribute caching completely
  • lookupcache=none: Disables positive lookup caching

Option 3: Application-Level Solution

For programmatic access, use stat() before reads:

# Python example
import os

def get_fresh_content(path):
    os.stat(path)  # Forces attribute refresh
    with open(path, 'r') as f:
        return f.read()

While noac gives you fresh data, it significantly increases server load. Consider these balanced approaches:

# Partial caching solution
server:/export /mnt/nfs nfs 
    rw,hard,intr,
    acregmin=0,acregmax=3,
    acdirmin=0,acdirmax=3 0 0

Monitor NFS operations in real-time:

# Server side
nfsstat -o all

# Client side
mount -o remount,verbose /mnt/nfs
tail -f /var/log/messages

The verbose mount option will show you exactly when attribute refreshes occur.