When using RealVNC/TightVNC between Windows Vista and Mac OS X 10.4.11, a peculiar freezing behavior occurs specifically during copy operations from the Vista host. The symptoms manifest as:

• Complete VNC window freeze (client-side)
• AppleVncServer process consuming 100MB+ RAM and 40-50% CPU
• Service becoming unresponsive until process restart

The root cause appears to be clipboard synchronization between the different VNC implementations. OS X Tiger's built-in VNC server (AppleVncServer) uses a different clipboard protocol than modern Windows VNC clients.

To verify this, try running from command line:

ps aux | grep AppleVNCServer
netstat -an | grep 5900

Option 1: Disable Clipboard Sync
Modify your VNC viewer config to disable clipboard synchronization:

# For TightVNC configuration file
Clipboard=0
SendClipboard=0
AcceptClipboard=0

Option 2: Alternative VNC Server
Consider replacing AppleVncServer with a more modern implementation:

# Install Vine Server (OSXvnc)
brew install --cask vine-server

Capture network traffic during the freeze event to identify the problematic clipboard packets:

tcpdump -i en0 -s 0 -w vnc_capture.pcap port 5900
# Analyze with Wireshark using 'vnc' filter

• Upgrade Mac OS X beyond 10.4.11 (Tiger's VNC implementation is outdated)
• Use SSH tunneling for better reliability: ssh -L 5900:localhost:5900 user@mac
• Consider modern alternatives like NoMachine or RDP

When using RealVNC/TightVNC between Vista and older Mac OSX (10.4.11), a peculiar freezing behavior emerges specifically during clipboard operations. The AppleVNCserver process balloons to 100MB+ RAM and 50% CPU while clients experience complete session freezes.

Here's what typically triggers it:
1. Successful VNC connection established
2. User performs copy operation on the Vista host (not within VNC session)
3. Within 60 seconds, VNC window freezes completely
4. Server process becomes unresponsive until manually restarted

# Sample process monitoring command on Mac
while true; do ps aux | grep AppleVNCServer; sleep 2; done

# Expected output before freeze:
user  1234  0.5  2.1  98732 104832  ??  S    2:45PM   0:45.42 /usr/sbin/AppleVNCServer

# After freeze:
user  1234  48.7 25.3  247852 253944 ??  R    2:47PM  12:31.88 /usr/sbin/AppleVNCServer

Option 1: Disable clipboard synchronization
In TightVNC settings (or registry):
HKCU\Software\TightVNC\Server\DisableClipboard=1

Option 2: Alternative VNC Server
Replace AppleVNCServer with more modern alternatives:

# On Mac via terminal:
curl -O https://example.com/vncserver.dmg
hdiutil attach vncserver.dmg
sudo installer -pkg /Volumes/VNC\ Server/Install.pkg -target /

Packet captures reveal the issue stems from RFB protocol version mismatches during clipboard updates. The Mac's legacy implementation doesn't properly handle certain clipboard encoding types from Windows Vista clients.

Wireshark filter to observe the issue:
tcp.port == 5900 && rfb

For those who must keep the original setup, add these parameters to /etc/applevnc.conf:

ClipboardMaxSize=1024
Protocol3.3Fallback=1
DisableExtendedClipboard=1

Then restart the service:
sudo killall AppleVNCServer