Windows 10 1809 RDP-UDP Transport Failure: Black Screen Issue on Reduced MTU Networks


3 views

After upgrading both RDP client and server to Windows 10 build 1809, users experience a black screen immediately following successful authentication when:

  • Operating over VPN connections with MTU ≤1350 bytes
  • UDP transport is enabled
  • Both endpoints run 1809 builds

The issue manifests specifically with the RDP-UDP Extensions v2 introduced in 1809 (MSDN ref: RDP-UDP Extensions v2). Packet captures show:

1. Normal TCP control channel establishment
2. UDP data channel initialization
3. Subsequent IP fragmentation (Wireshark ID: [fragment offset])
4. Failed reassembly attempts

Option 1: Disable UDP Transport

Group Policy configuration:

Computer Configuration > Administrative Templates > Windows Components > Remote Desktop Services > Remote Desktop Session Host > Connections
Set "Select RDP Transport Protocols" to "Use only TCP"

Option 2: Registry Modification

For systems without Group Policy:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\Terminal Services\Client]
"SelectTransport"=dword:00000001

For environments requiring UDP transport:

// PowerShell script to adjust interface MTU
Get-NetAdapter | Where-Object {$_.InterfaceDescription -match "VPN"} | 
Set-NetIPInterface -InterfaceMetric 4000 -NlMtuBytes 1400

While not officially acknowledged in Windows 10 release notes, this correlates with known issues in:

  • KB4483234 (pre-1903 builds)
  • RDP 10.2 protocol changes

The UWP RDP client (Microsoft Store version) demonstrates stable behavior as it:

  1. Defaults to TCP-only transport
  2. Implements different MTU detection logic
  3. Uses legacy RDP 8.1 protocol stack

After upgrading both client and server to Windows 10 1809, I encountered a peculiar RDP behavior where the session would display a black screen immediately after authentication when:

  • UDP transport is enabled
  • Connection traverses a reduced-MTU link (tested with VPN at ~1350 bytes)
  • Both endpoints run 1809 (older clients/servers work fine)

Network traces reveal the issue manifests after the initial TCP control channel handshake when UDP data transmission begins. The problematic flow shows:

1. [SYN], [SYN/ACK], [ACK] - TCP 3-way handshake
2. CredSSP negotiation over TCP
3. UDP channel establishment
4. Black screen coincides with:
   - IP fragments (UDP datagrams > path MTU)
   - Wireshark fails to reassemble (payload appears corrupt)

The introduction of RDP-UDP v2 in 1809 appears to mishandle path MTU discovery. Test cases confirm:

# Working scenarios:
- MTU 1500: Full functionality
- MTU ~1350 with TCP-only: Works
- Cross-version (1803→1809): Works

# Failure scenario:
- MTU ~1350 with UDP: Black screen

Option 1: Force TCP transport via Group Policy:

Path: Computer Configuration → Administrative Templates → Windows Components → Remote Desktop Services → Remote Desktop Session Host → Connections
Policy: "Select RDP transport protocols"
Set to: "Use only TCP"

Option 2: Adjust MTU settings (temporary VPN solution):

netsh interface ipv4 set subinterface "VPN" mtu=1400 store=persistent

Microsoft has acknowledged this in the Windows 10 Known Issues document. Two permanent fixes exist:

1. Apply KB4489899 (OS Build 17763.864+)

The update specifically addresses RDP-UDP fragmentation handling.

2. Registry modification for advanced users

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations]
"UDPTransportPayloadSize"=dword:00000500

This limits UDP datagrams to 1280 bytes (0x500 hex), safely below typical VPN MTUs.

The UWP RDP client (Microsoft Remote Desktop from Store) uses different network logic and works unaffected:

# PowerShell install command:
Get-AppxPackage Microsoft.RemoteDesktop | Remove-AppxPackage
Get-AppxPackage -AllUsers | Where Name -eq Microsoft.RemoteDesktop | Foreach {Add-AppxPackage -DisableDevelopmentMode -Register "$($_.InstallLocation)\AppXManifest.xml"}

For those implementing custom RDP solutions, test with this Python snippet to validate MTU handling:

import socket

def test_mtu(host, port=3389):
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    s.setsockopt(socket.IPPROTO_IP, socket.IP_MTU_DISCOVER, socket.IP_PMTUDISC_DO)
    try:
        s.connect((host, port))
        return s.getsockopt(socket.IPPROTO_IP, socket.IP_MTU)
    except socket.error as e:
        print(f"MTU discovery failed: {e}")
    finally:
        s.close()

print(f"Path MTU to server: {test_mtu('your_rdp_server')}")