When attempting to visualize OpenGL applications through VNC, many developers encounter the frustrating "Xlib: extension "GLX" missing on display" error. This occurs because traditional VNC implementations don't natively support hardware-accelerated OpenGL rendering.

VNC normally works by capturing the framebuffer and transmitting compressed screen updates. However, OpenGL rendering requires:

• Direct access to GPU hardware
• GLX extension for OpenGL/X11 integration
• Properly configured virtual framebuffer

For your RHEL 5 host with UltraVNC client, try these approaches:

1. VirtualGL + TurboVNC Combination

The most reliable method for remote OpenGL visualization:

# On RHEL 5 host:
sudo yum install VirtualGL turbojpeg
wget https://sourceforge.net/projects/turbovnc/files/turbovnc-2.2.5.tar.gz
tar xzf turbovnc-2.2.5.tar.gz
cd turbovnc-2.2.5
./configure
make
sudo make install

# Start VirtualGL server
/opt/VirtualGL/bin/vglserver_config
# Choose option 1 (Configure for use with VirtualGL)

# Launch TurboVNC session
/opt/TurboVNC/bin/vncserver -geometry 1920x1080

2. Xvfb with Software Rendering

For environments where GPU access isn't possible:

Xvfb :1 -screen 0 1280x1024x24 -extension GLX &
export DISPLAY=:1
# Your OpenGL application will now use software rendering

For UltraVNC on Windows XP:

• Enable JPEG compression in connection options
• Set color depth to 24-bit True Color
• Disable "Auto Select" for encoding method

When benchmarking remote OpenGL performance:

For modern applications that support EGL:

export MESA_GL_VERSION_OVERRIDE=3.3
export MESA_GLSL_VERSION_OVERRIDE=330
export LIBGL_ALWAYS_SOFTWARE=1

The error Xlib: extension "GLX" missing on display ":1.0" occurs because VNC servers typically don't implement GLX (OpenGL Extension to the X Window System). When your OpenGL application tries to create a rendering context, it fails because the virtual display doesn't support hardware-accelerated OpenGL.

There are two primary methods to handle OpenGL with VNC:

1. Software Rendering: Force OpenGL to use Mesa software rendering
2. VirtualGL: Offload rendering to server's GPU and transport pixels

For basic OpenGL applications, you can configure the environment to use software rendering:

# Start VNC server with mesa software rendering
vncserver :1 -geometry 1920x1080 -depth 24 \
  -extension GLX \
  -extension "Composite, RANDR, RENDER, DAMAGE, XFIXES" \
  -fp /usr/share/fonts/X11/misc/,/usr/share/fonts/X11/75dpi/,/usr/share/fonts/X11/100dpi/ \
  -co /etc/X11/rgb

# Environment variables for client
export LIBGL_ALWAYS_SOFTWARE=1
export GALLIUM_DRIVER=llvmpipe

For better performance with hardware acceleration:

# Server configuration
sudo yum install VirtualGL
vglserver_config

# Start VNC with VirtualGL
vncserver :1 -geometry 1920x1080 -depth 24
export DISPLAY=:1
vglrun glxinfo | grep "OpenGL"

Create a simple test script (test_gl.py):

import OpenGL
from OpenGL.GL import *
from OpenGL.GLUT import *

def display():
    glClear(GL_COLOR_BUFFER_BIT)
    glBegin(GL_TRIANGLES)
    glVertex2f(0, 1)
    glVertex2f(-1, -1)
    glVertex2f(1, -1)
    glEnd()
    glutSwapBuffers()

glutInit()
glutInitDisplayMode(GLUT_DOUBLE | GLUT_RGB)
glutCreateWindow(b"VNC OpenGL Test")
glutDisplayFunc(display)
glutMainLoop()

For modern RHEL systems using TigerVNC:

# /etc/systemd/system/vncserver@.service
[Service]
Type=simple
ExecStart=/usr/bin/vncserver %i -geometry 1920x1080 -depth 24 \
  -extension GLX -extension "Composite, RANDR, RENDER, DAMAGE, XFIXES" \
  -fp /usr/share/fonts/X11/misc/,/usr/share/fonts/X11/75dpi/,/usr/share/fonts/X11/100dpi/ \
  -co /etc/X11/rgb

For intensive graphics applications, consider:

• Using TurboVNC with VirtualGL for better compression
• Adjusting the JPEG compression quality (-quality 0-100)
• Enabling X Video Motion Compensation if available