When implementing Nginx as a reverse proxy for IIS with Windows Authentication (NTLM/Negotiate), we face several technical hurdles:

# Typical symptom in access logs
192.168.0.5 - - [09/Feb/2016:14:04:14 -0600] "GET / HTTP/1.1" 401 1293
192.168.0.5 - - [09/Feb/2016:14:04:31 -0600] "GET / HTTP/1.1" 401 341

The fundamental issue lies in NTLM's multi-step authentication protocol. Standard Nginx proxy configurations break the connection between authentication steps, causing endless 401 loops.

Key headers we observe in failed attempts:

WWW-Authenticate: NTLM
WWW-Authenticate: Negotiate
Authorization: NTLM TlRNTVMTUAAAB4IIAAAAAAAAAAAAACDAFGAAAAAAAAAAAAAAAAA=

The most reliable approach uses Nginx's stream module for TCP-level proxying:

# /etc/nginx/nginx.conf
stream {
    upstream iis_backend {
        server 192.168.0.20:80;
    }
    
    server {
        listen 443;
        proxy_pass iis_backend;
        proxy_connect_timeout 1s;
        proxy_timeout 3h; # Important for long NTLM sessions
    }
}

For cases where stream proxying isn't feasible, configure HTTP keepalive:

upstream iis_backend {
    server 192.168.0.20:80;
    keepalive 16; # Maintains connection for NTLM handshake
}

server {
    location / {
        proxy_pass http://iis_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        # Other standard proxy headers...
    }
}

On the IIS server, ensure these settings:

1. Enable "Extended Protection" = Off (for testing)
2. Add your proxy server's IP to "Back Connection Host Names"
3. Configure SPN if using Kerberos

When debugging, check these key points:

Here's a production-tested configuration combining both approaches:

# For HTTP traffic
upstream iis_http {
    server 192.168.0.20:80;
    keepalive 32;
}

# For HTTPS traffic
stream {
    upstream iis_https {
        server 192.168.0.20:443;
    }
    
    server {
        listen 443;
        proxy_pass iis_https;
        proxy_timeout 1h;
    }
}

When setting up Nginx as a reverse proxy for IIS with Windows Authentication (NTLM/Negotiate), the main obstacle is maintaining the connection state required for the multi-step authentication handshake. The 401 Unauthorized loops you're experiencing occur because Nginx's default HTTP/1.0 proxy behavior breaks the NTLM authentication flow.

For successful NTLM passthrough, your Nginx configuration must:

• Use HTTP/1.1 for persistent connections
• Preserve all authentication headers
• Maintain connection state between client and backend

Here's the tested configuration that resolves the authentication loop:

# /etc/nginx/nginx.conf
worker_processes auto;
events {
    worker_connections 1024;
}

http {
    upstream iis_backend {
        server 192.168.0.20:80;
        keepalive 32;
    }

    server {
        listen 443 ssl;
        server_name server2.mydomain.com;

        ssl_certificate /path/to/cert.pem;
        ssl_certificate_key /path/to/key.pem;
        
        location / {
            proxy_pass http://iis_backend;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            
            # Critical for NTLM
            proxy_set_header Authorization $http_authorization;
            proxy_pass_request_headers on;
            proxy_set_header Connection keep-alive;
            
            # Timeout adjustments
            proxy_connect_timeout 300s;
            proxy_read_timeout 300s;
        }
    }
}

1. Upstream Keepalive: The keepalive 32 directive maintains a pool of persistent connections to IIS, which is essential for NTLM's multi-step authentication.

2. HTTP/1.1 Protocol: proxy_http_version 1.1 enables connection persistence that NTLM requires.

3. Header Preservation: The proxy_set_header Authorization directive ensures authentication tokens aren't stripped during proxying.

For complex scenarios, Nginx's stream module can provide a TCP-level proxy that completely bypasses HTTP header issues:

stream {
    upstream iis_tcp {
        server 192.168.0.20:80;
    }

    server {
        listen 443;
        proxy_pass iis_tcp;
        proxy_protocol on;
    }
}

If issues persist:

1. Verify Nginx version is 1.9.9 or newer
2. Check that keepalive is properly configured in both upstream and location blocks
3. Ensure IIS is configured to allow authentication from proxy IPs
4. Inspect both Nginx and IIS logs simultaneously during authentication attempts

When using NTLM through a reverse proxy:

• Increase worker_connections to handle more concurrent auth attempts
• Adjust timeouts to accommodate slower authentication processes
• Consider implementing caching for static resources to reduce authentication overhead