When working with NFS (Network File System) shares on RHEL5 systems, you might encounter mysterious files named .nfsXXXX (where XXXX represents hexadecimal characters). These files typically appear when:

• An application has open file handles on the NFS server
• Processes are abruptly terminated while accessing files
• Files are being moved/renamed while still in use

Based on your description of an application streaming data to NFS with hourly file movements, these files likely occur because:

// Typical sequence causing .nfs files:
1. Application opens "hourly_data.log" for writing
2. Writes data continuously
3. Attempts to rename/move the file while still holding the handle
4. NFS client creates .nfsXXXX as a placeholder

The .nfsXXXX files are created by the NFS client to maintain Unix file semantics when:

• A file is deleted while still open (NFS "silly rename" behavior)
• The NFS server can't guarantee immediate file removal
• Client-side caching conflicts with server operations

To investigate the issue:

# Check open files on the NFS server
lsof | grep nfs | grep deleted

# Find stale .nfs files older than 1 day
find /nfs/mount/point -name ".nfs*" -mtime +1

# Monitor NFS operations in real-time
nfsstat -c  # Client statistics
nfsstat -s  # Server statistics

Option 1: Application-level fixes

// Before renaming/moving files:
1. flush() all writes
2. close() the file handle
3. Then perform rename operation

// Python example:
with open('hourly.log', 'w') as f:
    f.write(data)
    f.flush()
os.rename('hourly.log', 'archive/hourly_20230801.log')

Option 2: NFS Configuration

# In /etc/exports on NFS server:
/nfs/share client.ip(rw,sync,no_subtree_check,no_wdelay)

# Client mount options in /etc/fstab:
server:/share /mnt nfs rw,hard,intr,noac 0 0

Option 3: Cleanup Script

#!/bin/bash
# Safely remove stale .nfs files
find /nfs/mount -name ".nfs*" -type f -mtime +7 | while read file; do
    if ! lsof "$file" >/dev/null; then
        rm -f "$file"
    fi
done

• Implement proper file handle management in your application
• Consider using fsync() before rename operations
• Monitor NFS latency and timeouts
• Use NFSv4 instead of NFSv3 if possible
• Consider alternative protocols like SMB or CIFS for heavy write workloads

When working with NFS-mounted directories in Linux environments (particularly RHEL/CentOS systems), you may encounter mysterious files with patterns like .nfsa1b2 or .nfs7f3e. These are temporary files created by the NFS client subsystem when:

• A process has a file open for writing
• The original file gets deleted while still being held open
• The application is actively writing to the file during rename/move operations

The NFS protocol (especially versions 2/3) doesn't properly handle the Unix behavior of allowing processes to keep writing to deleted files. When this occurs, the Linux NFS client creates these hidden temporary files to maintain data integrity. Here's what's happening under the hood:

// Simplified sequence triggering .nfsXXXX creation
1. Process opens file.dat (file descriptor #42)
2. Process writes to fd #42
3. Another process (or same process) executes: rename("file.dat", "archive.dat")
4. NFS client creates .nfsXXXX to hold the still-open file's data
5. Original file.dat appears deleted but remains accessible to process holding fd #42

For the described scenario where hourly files are moved after creation, we can use these diagnostic commands:

# Check for processes holding deleted files
lsof +L1 /nfs/mount/point

# Monitor NFS operations in real-time
nfsstat -c  # Client statistics
cat /proc/fs/nfsfs/servers  # Detailed NFS client state

Immediate mitigation:

# Safely clean up stale .nfsXXXX files (ensure no processes are using them first):
find /nfs/mount/point -name '.nfs*' -mtime +1 -delete

Code-level fixes for the application:

// Python example: Safe file rotation pattern
import os

def write_hourly_data(filename):
    # Write to temp file first
    tempname = f"{filename}.{os.getpid()}"
    with open(tempname, 'w') as f:
        f.write(hourly_data)
    
    # Atomic rename (NFSv3 compatible)
    os.rename(tempname, filename)

# Alternative for NFSv4+ deployments:
def safe_rotate(source, target):
    try:
        os.link(source, target)
        os.unlink(source)
    except OSError:
        os.rename(source, target)

The behavior varies by NFS version and mount options. These mount options can help:

# /etc/fstab entry with stability improvements
nfs-server:/export /mnt nfs 
    rw,hard,intr,noac,noatime,nodiratime,actimeo=3 0 0

Key options explanation:

• hard: Prevents silent data corruption
• noac: Bypasses attribute caching (helps with consistency)
• actimeo=3: Reduces cache time for file attributes

For critical applications, consider:

• Using local storage with rsync for final file placement
• Implementing a message queue (RabbitMQ/Kafka) instead of file-based IPC
• Switching to SMB/CIFS if Windows clients are involved
• Upgrading to NFSv4.1+ with pNFS for better consistency