When implementing Windows Authentication in IIS 7.5, the server initiates a specific challenge-response sequence:

1. Client requests protected resource
2. Server responds with HTTP 401 (Unauthorized)
3. Browser shows credential dialog
4. Client resends request with Authorization header
5. Server grants access or repeats 401

The critical issue occurs at step 2 - IIS intercepts its own 401 response when custom error pages are configured, preventing the NTLM/Negotiate handshake from completing.

Here's a functional web.config snippet that preserves authentication while showing custom errors:

<configuration>
  <system.webServer>
    <security>
      <authentication>
        <windowsAuthentication enabled="true" />
      </authentication>
    </security>
    <httpErrors errorMode="DetailedLocalOnly">
      <remove statusCode="401" subStatusCode="2" />
      <error statusCode="401" 
             subStatusCode="2" 
             path="/errors/401.htm" 
             responseMode="File" />
    </httpErrors>
  </system.webServer>
</configuration>

Notice these critical elements that differ from typical error page setups:

• responseMode="File": Uses file path instead of URL execution
• errorMode="DetailedLocalOnly": Shows custom errors only to remote clients
• Separate error directory: /errors/ location outside protected areas

For better user experience, add JavaScript to your 401.htm that detects authentication failures:

<script>
if (window.performance && performance.navigation.type === 2) {
    document.getElementById('retry-section').style.display = 'block';
}
</script>

Verify your configuration works through these test cases:

1. First access attempt shows Windows auth dialog
2. Three failed attempts show your custom page
3. Successful login bypasses the error page
4. Direct access to /errors/401.htm works without auth

For more control, consider using URL Rewrite as a post-authentication handler:

<rule name="Auth Failed Redirect" enabled="true" stopProcessing="true">
  <match url=".*" />
  <conditions>
    <add input="{HTTP_X_AUTH_ERROR}" pattern="1" />
  </conditions>
  <action type="Redirect" url="/errors/authfailed" appendQueryString="false" />
</rule>

When implementing Windows Authentication in IIS 7.5, the standard 401 error page appears when users cancel or fail authentication. While we want to replace this with a custom page, simply configuring error page redirection breaks the authentication flow entirely. The browser never shows the credential prompt because IIS immediately redirects to the custom error page instead of completing the authentication handshake.

Windows Authentication works through a challenge-response mechanism:

1. IIS sends 401 Unauthorized response
2. Browser displays credential prompt
3. User submits credentials
4. Browser resends request with Authorization header

Our custom error page configuration interrupts this at step 1.

Instead of using IIS's error page redirection, we'll implement this at the application level:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <security>
            <authentication>
                <windowsAuthentication enabled="true" />
            </authentication>
        </security>
        <httpErrors errorMode="DetailedLocalOnly">
            <remove statusCode="401" />
        </httpErrors>
    </system.webServer>
    <system.web>
        <customErrors mode="RemoteOnly" defaultRedirect="~/Error.aspx">
            <error statusCode="401" redirect="~/Unauthorized.aspx" />
        </customErrors>
        <authentication mode="Windows" />
        <authorization>
            <deny users="?" />
        </authorization>
    </system.web>
</configuration>

For PHP applications, we can check the authentication status and redirect manually:

<?php
if (!isset($_SERVER['PHP_AUTH_USER'])) {
    header('WWW-Authenticate: NTLM');
    header('HTTP/1.0 401 Unauthorized');
    include('custom_401.php');
    exit;
} else {
    // Process authenticated request
}
?>

Create a rule that only redirects after failed authentication attempts:

<rule name="Custom 401 Redirect" enabled="true" stopProcessing="true">
    <match url=".*" />
    <conditions>
        <add input="{HTTP_X_AUTHORIZATION}" pattern="^NTLM" negate="true" />
        <add input="{HTTP_AUTHORIZATION}" pattern="^Basic" negate="true" />
        <add input="{RESPONSE_STATUS}" pattern="^401" />
    </conditions>
    <action type="Redirect" url="/custom_401.html" appendQueryString="false" />
</rule>

• Keep the custom error page in an unauthenticated location
• Test with different browsers as they handle NTLM differently
• Consider adding clear instructions for domain users
• Log authentication failures for security monitoring