When dealing with virtual machine disk images stored on remote NFS shares, IO performance becomes critical. My testing showed a significant gap between local OpenSolaris writes (1.6GB/s) and remote CentOS writes (50MB/s) despite having direct gigE connectivity with jumbo frames (MTU=9000) configured.

Before diving into NFS tuning, let's verify network fundamentals:

# On both systems check MTU settings
ifconfig | grep mtu

# Test basic network throughput (install iperf if needed)
iperf -c nfs-server-ip -t 20 -i 2 -f m

First, optimize the OpenSolaris NFS server:

# Increase NFS server threads (default is often too low)
sharectl set -p server_versmax=4 nfs
sharectl set -p server_versmin=4 nfs
sharectl set -p server_delegation=on nfs
sharectl set -p nfsmapid_domain=yourdomain.com nfs

For the specific share, consider these export options:

share -F nfs -o rw,async,no_root_squash,anon=0 /path/to/vmstorage

On the CentOS client, mount options make a huge difference:

# In /etc/fstab
nfs-server:/path/to/vmstorage /mnt/vmstorage nfs 
rw,hard,intr,rsize=65536,wsize=65536,noatime,nodiratime,async,proto=tcp,timeo=600 0 0

Additional sysctl tweaks for the client:

# Add to /etc/sysctl.conf
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

For VM workloads specifically:

• Consider NFSv4.1 with pNFS if supported
• Test with different block sizes (dd if=/dev/zero of=testfile bs=1M count=1024)
• Monitor with nfsiostat -s 5

After changes, verify performance:

# Write test
dd if=/dev/zero of=/mnt/nfs/testfile bs=1M count=1024 conv=fdatasync

# Read test (clear cache first)
echo 3 > /proc/sys/vm/drop_caches
dd if=/mnt/nfs/testfile of=/dev/null bs=1M count=1024

For continuous monitoring:

nfsstat -cn
nfsstat -sn
vmstat 2 10

In my infrastructure, CentOS 5 VMs running on VMware are accessing disk images stored on an OpenSolaris 2009.06 server via NFS. Initial benchmarks show significant performance disparity:

# Local OpenSolaris write test:
dd if=/dev/zero of=/nfsstorage/testfile bs=1024k count=400
# Result: ~1.6GB/s

# Remote CentOS write test:
dd if=/dev/zero of=/mnt/nfs/testfile bs=1024k count=400
# Result: ~50MB/s

The systems are directly connected via gigabit Ethernet with jumbo frames enabled (MTU=9000). Current NFS mount options are using defaults, which is clearly suboptimal for VM workloads.

First, let's optimize the NFS server configuration:

# Increase NFS server threads
echo "max_nfs_threads=64" >> /etc/system

# Adjust TCP parameters
ndd -set /dev/tcp tcp_recv_hiwat 524288
ndd -set /dev/tcp tcp_xmit_hiwat 524288

# Tune NFS server parameters
share -F nfs -o rw,async,no_root_squash,no_wdelay /export/vmstorage

For the Linux NFS client, we need to modify mount options:

# /etc/fstab entry example:
opensolaris:/export/vmstorage /mnt/nfs nfs rw,hard,intr,noatime,nodiratime,rsize=32768,wsize=32768,tcp,timeo=600 0 0

Both systems benefit from TCP stack tuning:

# On CentOS (add to /etc/sysctl.conf):
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

After applying changes, verify performance with proper tools:

# Use nfsiostat for NFS-specific metrics
nfsiostat -d 5

# Network monitoring
nicstat -i eth0 5

For environments with multiple VMs, consider:

• Implementing NFSv4 with parallel NFS (pNFS)
• Testing different congestion control algorithms (try cubic or bbr)
• Evaluating RDMA over Converged Ethernet (RoCE) for ultra-low latency