When working with ZFS-backed storage over 1Gb Ethernet, these are typical performance benchmarks you should expect:

• iSCSI (Windows 2008/NTFS): 90-110MB/s sequential read/write (with proper TCP offload)
• NFS (VMware/CentOS): 80-100MB/s with NFSv3, 90-110MB/s with NFSv4

For OpenSolaris ZFS shares, these tunables significantly impact performance:

# iSCSI target settings
iscsi_max_recv_data_segment_length=262144
iscsi_max_send_data_segment_length=262144

# NFS server tuning
set sharectl nfs -o server_vers=4,server_delegation=on
echo "set zfs:zfs_nocacheflush=1" >> /etc/system

Since you're using direct connections (no switches), consider these adjustments:

# Jumbo frames configuration (both endpoints)
ifconfig ethX mtu 9000

# TCP stack tuning (OpenSolaris)
ndd -set /dev/tcp tcp_max_buf 4194304
ndd -set /dev/tcp tcp_xmit_hiwat 1048576
ndd -set /dev/tcp tcp_recv_hiwat 1048576

Your munin graphs should show:

• Consistent 800-950Mbps utilization during sequential transfers
• Latency under 2ms for local direct connections
• Packet error rate below 0.001%

For your CentOS/VMware setup, add these ESXi advanced parameters:

NFS.MaxVolumes = 256
NFS.HeartbeatDelta = 15
NFS.MaxQueueDepth = 64
Net.TcpipHeapSize = 128
Net.TcpipHeapMax = 1536

On your Windows 2008 server, apply these registry changes:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
"TcpWindowSize"=dword:00100000
"Tcp1323Opts"=dword:00000003

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management]
"LargeSystemCache"=dword:00000001

Common issues and their solutions:

When working with 1Gb Ethernet storage protocols, you should expect these theoretical maximums:

• Raw 1GbE throughput: 125 MB/s (after overhead ~112-118 MB/s)
• iSCSI (TCP/IP overhead): 90-110 MB/s sustained
• NFSv3 (UDP): 80-100 MB/s
• NFSv4 (TCP): 95-110 MB/s

From your munin graphs, I notice:

# Sample munin output interpretation
NFS Client stats show:
- Average read: ~45MB/s
- Average write: ~30MB/s
- Peak throughput: ~65MB/s

These numbers suggest room for improvement. Let's examine tuning parameters.

For OpenSolaris iSCSI target (COMSTAR):

# Adjust queue depths
iscsitadm modify target -p 32 targetname
iscsitadm modify tpg -p 16 tpname

# Windows 2008 initiator settings (PowerShell):
Set-MSDSMSupportedHW -VendorId MSFT2005 -ProductId iSCSIBusType_0x9 -MaxIO 32
Set-MSDSMGlobalDefault -MaxRequestBlocks 4096

For VMware workloads, these ZFS/NFS settings help:

# On OpenSolaris:
zfs set primarycache=metadata tank/vmware
zfs set recordsize=8k tank/vmware
zfs set sync=disabled tank/vmware # Only for non-critical VMs!

# NFS server tuning:
sharectl set -p max_connections=1000 nfs
sharectl set -p server_versmax=4 nfs

Since you're using direct connections:

# Jumbo frames (if supported):
ifconfig e1000g0 mtu 9000

# Disable flow control (direct connect):
dladm set-linkprop -p flowctrl=off e1000g0

# Interrupt moderation:
ndd -set /dev/e1000g0 intr_throttling 8000

Here's what I've measured in similar setups:

Beyond munin, consider:

# Solaris performance tools:
kstat -n e1000g0 -p | grep bytes64
dtrace -n 'io:::start { @bytes = quantize(args[0]->b_bcount); }'

# Windows iSCSI perf:
Get-Counter '\MSiSCSI Portal(*)\Bytes/sec'