When dealing with UDP-based services in .NET, traditional load balancing approaches often fall short. Unlike HTTP where we have rich middleware options, UDP requires more specialized handling - especially when you need both load distribution and session affinity (client stickiness). Here's how to solve this for a stateless data collection service.

Microsoft's Network Load Balancing (NLB) does provide basic UDP load balancing with client affinity, but as you've discovered, it broadcasts ARP requests across the network segment. For production environments, this can cause unnecessary noise. A better approach would be implementing a custom solution at the application layer.

Consider building a lightweight UDP proxy that implements consistent hashing for session affinity. Here's a basic implementation:

public class UdpLoadBalancer
{
    private readonly UdpClient _listener;
    private readonly Dictionary<IPEndPoint, UdpClient> _clientMap;
    private readonly List<ServerEndpoint> _servers;
    private readonly object _lock = new object();

    public UdpLoadBalancer(int listenPort, List<ServerEndpoint> servers)
    {
        _listener = new UdpClient(listenPort);
        _servers = servers;
        _clientMap = new Dictionary<IPEndPoint, UdpClient>();
    }

    public async Task Start()
    {
        while (true)
        {
            var result = await _listener.ReceiveAsync();
            var clientEndpoint = result.RemoteEndPoint;
            
            lock (_lock)
            {
                if (!_clientMap.TryGetValue(clientEndpoint, out var serverClient))
                {
                    // Use consistent hashing to assign server
                    var serverIndex = GetConsistentHash(clientEndpoint) % _servers.Count;
                    serverClient = new UdpClient();
                    serverClient.Connect(_servers[serverIndex].Address, _servers[serverIndex].Port);
                    _clientMap[clientEndpoint] = serverClient;
                }
                
                await serverClient.SendAsync(result.Buffer, result.Buffer.Length);
            }
        }
    }

    private int GetConsistentHash(IPEndPoint endpoint)
    {
        // Simple hash implementation - consider using better algorithm
        return endpoint.Address.GetHashCode() ^ endpoint.Port;
    }
}

For enterprise scenarios, consider these alternatives:

• HAProxy: Configured with 'source' load balancing algorithm for UDP (needs version 1.8+)
• NGINX Plus: Commercial version supports UDP load balancing with session persistence
• Envoy Proxy: Offers advanced UDP load balancing capabilities

Since you mentioned no DNS is available, we'll skip round-robin DNS solutions. However, if DNS becomes an option later, you could implement a hybrid approach where the DNS returns multiple IPs but the client implements its own consistent hashing.

Whatever solution you choose, implement proper health checking:

public async Task<bool> IsServerHealthy(ServerEndpoint server)
{
    try 
    {
        using var client = new UdpClient();
        client.Connect(server.Address, server.Port);
        await client.SendAsync(new byte[1], 1);
        return (await client.ReceiveAsync().WaitAsync(TimeSpan.FromSeconds(1))).Buffer.Length > 0;
    }
    catch
    {
        return false;
    }
}

When a server fails, you'll need to redistribute its clients. The consistent hashing approach makes this relatively straightforward - simply remove the failed node from your server list and existing clients will naturally redistribute based on the new server count.

When dealing with UDP-based systems in production environments, load balancing presents unique challenges compared to TCP. The stateless nature of UDP requires careful consideration when distributing traffic across multiple server instances while maintaining session affinity (also called client stickiness).

Microsoft NLB (Network Load Balancing) can work for basic scenarios, but as you've observed, it broadcasts traffic to all cluster nodes, creating network noise. For UDP applications, consider these alternatives:

// Example of hashing client IP for consistent routing
public class UdpRouter
{
    private readonly List _servers;
    
    public ServerEndpoint GetTargetServer(IPEndPoint clientEndpoint)
    {
        uint hash = (uint)clientEndpoint.Address.GetHashCode();
        return _servers[(int)(hash % _servers.Count)];
    }
}

For custom protocols, implementing a dispatcher service often provides better control:

// Dispatcher service using consistent hashing
public class UdpDispatcher
{
    private readonly ConcurrentDictionary _clientMappings 
        = new ConcurrentDictionary();
    private readonly List _availableServers;
    private readonly object _balanceLock = new object();
    
    public ServerEndpoint RouteClient(IPEndPoint clientEndpoint)
    {
        string clientKey = clientEndpoint.ToString();
        
        if (_clientMappings.TryGetValue(clientKey, out var existing))
        {
            return existing;
        }
        
        lock (_balanceLock)
        {
            // Find least loaded server
            var target = _availableServers
                .OrderBy(s => s.CurrentLoad)
                .First();
            
            _clientMappings[clientKey] = target;
            return target;
        }
    }
}

Combine IP-based routing with server health monitoring:

// Health-aware UDP load balancer
public class HealthAwareUdpBalancer : IHealthChecker
{
    private readonly List _servers = new List();
    private readonly Timer _healthTimer;
    
    public HealthAwareUdpBalancer()
    {
        _healthTimer = new Timer(CheckServerHealth, null, 
            TimeSpan.FromMinutes(1), 
            TimeSpan.FromMinutes(1));
    }
    
    private void CheckServerHealth(object state)
    {
        Parallel.ForEach(_servers, server => 
        {
            server.IsHealthy = PingServer(server);
            server.CurrentLoad = GetServerLoad(server);
        });
    }
    
    public ServerEndpoint GetHealthyServer(IPEndPoint client)
    {
        var healthyServers = _servers.Where(s => s.IsHealthy).ToList();
        if (!healthyServers.Any()) throw new NoHealthyServersException();
        
        // Implementation of consistent hashing or round-robin
        return SelectServer(client, healthyServers);
    }
}

• Implement circuit breakers for failed servers
• Add logging for routing decisions
• Consider using UDP proxy solutions like Envoy or HAProxy
• Monitor packet loss between balancer and servers

For complex scenarios, building a dedicated UDP proxy can offer better control:

public class UdpProxyWorker
{
    private readonly UdpClient _listener;
    private readonly List _servers;
    
    public async Task StartAsync()
    {
        while (true)
        {
            var result = await _listener.ReceiveAsync();
            var target = _router.GetTargetServer(result.RemoteEndPoint);
            
            await SendToServer(target, result.Buffer);
        }
    }
    
    private async Task SendToServer(ServerEndpoint target, byte[] data)
    {
        using (var client = new UdpClient())
        {
            await client.SendAsync(data, data.Length, target.Endpoint);
        }
    }
}

Remember to benchmark different approaches under expected load conditions. The optimal solution depends on your specific protocol characteristics and server capabilities.