Optimizing Slow MSMQ Message Receiving in High-Volume Clustered Environments


4 views

In our Windows Server 2008 R2 environment with VMware vSphere 4.0 U1, we observed severe MSMQ performance degradation where:

  • 8 web servers (2 vCPUs, 2GB RAM) couldn't push messages beyond 2/sec
  • Clustered MSMQ servers (4 vCPUs, 8GB RAM) received only ~7 messages/sec
  • Message backlog reached 300K in outbound queues and 1M+ in cluster queues

First, verify these performance counters:

# PowerShell to check MSMQ performance
Get-Counter "\MSMQ Queue(*)\Messages in Queue"
Get-Counter "\MSMQ Service\Messages Received/sec"
Get-Counter "\MSMQ Service\Messages Sent/sec"

Key observations from our environment:

# Sample output from a problematic queue
Path                                                        InstanceName      CookedValue
----                                                        ------------      -----------
\\SERVER01\MSMQ Queue(webcluster$\processing)\Messages in Queue _total          1048576

For clustered MSMQ environments, apply these registry optimizations:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSMQ\Parameters]
"TCPNoDelay"=dword:00000001
"MessageBufferSize"=dword:00004000
"SocketBufferSize"=dword:00008000

In virtualized environments, pay special attention to:

  • VMXNET3 network adapters instead of E1000
  • Disable TCP Chimney Offload: netsh int tcp set global chimney=disabled
  • Enable Jumbo Frames on vSwitches (9000 MTU)

When receiving messages programmatically:

// C# optimized message receiver
var queuePath = @".\private$\processing"; 
using (var queue = new MessageQueue(queuePath))
{
    queue.Formatter = new BinaryMessageFormatter();
    queue.DefaultPropertiesToSend.UseAuthentication = false;
    queue.DefaultPropertiesToSend.UseJournalQueue = false;
    
    // Critical performance settings
    queue.SetPermissions("Everyone", MessageQueueAccessRights.FullControl);
    queue.MessageReadPropertyFilter.SetAll();
    
    while (true)
    {
        try 
        {
            // Async peek is crucial for performance
            var message = queue.Receive(TimeSpan.FromSeconds(1));
            // Process message
        }
        catch (MessageQueueException mqex)
        {
            if (mqex.MessageQueueErrorCode != MessageQueueErrorCode.IOTimeout)
                throw;
        }
    }
}

For Windows Server clusters hosting MSMQ:

  1. Set HKLM\Cluster\LogLevel to 3 for detailed logging
  2. Disable SMB signing: Set-SmbClientConfiguration -RequireSecuritySignature $false
  3. Adjust CSV cache: Set-ClusterParameter -Name CsvBlockCacheSize -Value 512

After applying changes, monitor with:

# PowerShell validation script
$counters = @(
    "\MSMQ Service\Messages Received/sec",
    "\PhysicalDisk(*)\Avg. Disk sec/Transfer",
    "\Processor(*)\% Processor Time"
)
Get-Counter -Counter $counters -SampleInterval 5 -MaxSamples 12

In our virtualized infrastructure running vSphere 4.0 Update 1, we encountered severe MSMQ performance degradation with the following characteristics:

// Typical perfmon counters showing the issue
MSMQ Service\\Messages Sent/sec = ~2
MSMQ Service\\Messages Received/sec = ~7
Application Servers\\Messages Processed/sec = 0-12

The environment consists of:

  • 8 Windows 2008 Web Edition VMs (2vCPU/2GB RAM) as front-end servers
  • 1 Windows 2008 Enterprise Edition clustered MSMQ server (4vCPU/8GB RAM)
  • 4 application servers processing messages

When we noticed queues backing up (300K messages in outbound, 1M+ in central queues), we:

  1. Verified disk space (46GB free of 50GB)
  2. Upgraded PowerPath to latest version
  3. Increased cluster nodes from 2vCPU to 4vCPU
  4. Checked network connectivity between VLANs

Several approaches helped improve throughput:

// PowerShell script to adjust MSMQ performance settings
Set-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\MSMQ\Parameters" -Name "TCPNoDelay" -Value 1
Set-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\MSMQ\Parameters" -Name "MessageDeadletter" -Value 0
Set-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\MSMQ\Parameters" -Name "EnableCache" -Value 1

For clustered MSMQ servers, additional tuning is needed:

  • Set HKLM\Cluster\MSMQ\Parameters\EnableTriggerNotification to 0
  • Configure HKLM\Cluster\MSMQ\Parameters\ConnectionCacheSize to 5000
  • Adjust HKLM\System\CurrentControlSet\Services\MSMQ\Parameters\JournalQuota to 10MB

Virtualization adds complexity to MSMQ performance:

esxcli system settings advanced set -o /Net/FollowHardwareMac -i 1
esxcli system settings advanced set -o /Net/UseHwTSO -i 1
esxcli system settings advanced set -o /Net/TcpipDefAckDelay -i 1

Create a comprehensive monitoring solution with these key counters:

// Sample SCOM or custom monitoring thresholds
MSMQ Queue\\Bytes in Queue > 50MB = Warning
MSMQ Service\\Messages Received/sec < 100 = Critical
MSMQ Queue\\Messages in Queue > 10,000 = Critical