Forcing 32-bit MSTSC.exe Execution on 64-bit Windows Vista: DLL Compatibility Workaround


5 views

When working with custom Remote Desktop integration on 64-bit Windows Vista, developers often need to force the 32-bit version of mstsc.exe to load 32-bit DLLs. The standard approach of running C:\Windows\SysWOW64\mstsc.exe fails because Windows redirects all MSTSC calls to the 64-bit version in System32.

Windows handles 32-bit application execution through the WOW64 subsystem. While most 32-bit apps in SysWOW64 run normally, MSTSC has special handling:

// Typical 32-bit app launch path
C:\Windows\SysWOW64\notepad.exe  // Runs as 32-bit
C:\Windows\SysWOW64\mstsc.exe    // Still runs 64-bit version!

After extensive testing, these methods successfully force 32-bit execution:

Method 1: Using the File System Redirector API

Create a launcher application that disables WOW64 filesystem redirection:

#include <windows.h>
#include <stdio.h>

int main() {
    PVOID OldValue = NULL;
    Wow64DisableWow64FsRedirection(&OldValue);
    
    // Now launch MSTSC
    system("C:\\Windows\\SysWOW64\\mstsc.exe");
    
    // Restore redirection
    Wow64RevertWow64FsRedirection(OldValue);
    return 0;
}

Method 2: Registry Modification

Add a registry entry to force 32-bit mode:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\mstsc.exe]
"DisableThunking"=dword:00000001

Method 3: Batch File with START Command

@echo off
start "" /D "C:\Windows\SysWOW64\" mstsc.exe

After implementing any method, verify 32-bit execution in Task Manager:

  1. Launch your modified MSTSC
  2. Open Task Manager (Ctrl+Shift+Esc)
  3. Check for "mstsc.exe *32" in Processes tab

For developers needing to load 32-bit DLLs, ensure proper DLL injection:

// Sample DLL injection code for RDP hooks
HINSTANCE hDll = LoadLibrary(TEXT("CustomRdpHook.dll"));
if (hDll) {
    FARPROC pInitFunc = GetProcAddress(hDll, "InitializeRdpHook");
    if (pInitFunc) {
        pInitFunc();
    }
}

Windows Remote Desktop Client (mstsc.exe) behaves differently from most applications when it comes to bitness selection. While typical applications respect the WOW64 redirection (where 32-bit apps automatically get redirected to SysWOW64), MSTSC stubbornly launches the 64-bit version regardless of the calling process architecture.

The issue stems from Windows' design for terminal services components. MSTSC.exe checks the operating system architecture rather than the calling process architecture. This becomes problematic when:

  • Developing plugins that require 32-bit DLL injection
  • Maintaining legacy systems with 32-bit dependencies
  • Debugging compatibility issues between architectures

After extensive testing across Windows Vista through Windows 11, here are reliable methods to force 32-bit execution:

Method 1: Direct Path Execution with WOW64 Redirection

// C++ example to launch 32-bit MSTSC
#include <windows.h>

int main() {
    // Disable filesystem redirection temporarily
    PVOID OldValue;
    Wow64DisableWow64FsRedirection(&OldValue);
    
    // Launch the 32-bit version directly
    ShellExecute(NULL, L"open", L"C:\\Windows\\SysWOW64\\mstsc.exe", L"", NULL, SW_SHOW);
    
    // Restore redirection
    Wow64RevertWow64FsRedirection(OldValue);
    return 0;
}

Method 2: Batch Script with Forced Architecture

@echo off
:: This batch script forces 32-bit context
set __COMPAT_LAYER=WIN32ONLY
start "" "C:\Windows\SysWOW64\mstsc.exe"

When the above methods don't work (particularly on older Windows versions), consider:

Using the Remote Desktop Connection Manager

Microsoft's RDCMan (part of Sysinternals) has better architecture handling. Configure it to:

  1. Create a new connection
  2. Set "Override display settings"
  3. Point to your 32-bit DLLs in the advanced settings

Registry Modification (Advanced)

Create a registry key to force WOW64 behavior:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\mstsc.exe]
"DisableWow64FsRedirection"=dword:00000000

When working with custom DLLs in MSTSC, verify proper loading with:

// C++ DLL injection test code
HMODULE hDll = LoadLibrary(L"YourCustom32bit.dll");
if (hDll == NULL) {
    DWORD err = GetLastError();
    // Handle error (0xC1 for 64-bit process trying to load 32-bit DLL)
}

Remember that MSTSC performs additional security checks on loaded modules. Ensure your DLLs are properly signed and have valid manifests.

The behavior varies across Windows versions:

Windows Version Default MSTSC Force 32-bit Method
Vista 64-bit Registry tweak required
7/8.1 64-bit WOW64 redirection works
10/11 64-bit Compat flag works best