SPN Hell

Ran across an interesting problem today.

It started with the classic symptoms of an IIS App Pool running under a domain identity. The one where you get the HTTP 401.1 errors and have to have a Domain Administrator log in and set the SPN for the app pool [1].

But there was a bit of extra weirdness in there. Turns out that it failed (with above symptoms) for all but one person. We spent some time working on that, but were unable to figure out what magic properties he had that allowed him in.

After rebooting the server, no one could get in. So the mystery of the magic privileges was solved.

So then we looked to verify the credentials that had been entered into the app pool. Sure enough, there was a typo in there. Reenter the password, reboot.

Now it works for everyone EXCEPT the person who had previously been the only one who could access it. Curiouser and curiouser. . . .

Chipping away at that for a while. We discover that he can access it by IP address no problem, and he can access it by name from other boxes. So it's not his credentials, and since he can ping the box by name, it's not DNS or connectivity. I'm guessing something that smells like Kerberos on his machine.

Using the fabulous tool ieHTTPHeaders [2], we can see that he is definitely passing a Kerberos ticket. It's not falling back to NTLM or anything like that.

We grab some stuff from Microsoft and take a look at his tickets. Then in a fit of rage, just purge them with KList. Suddenly it all starts working. Hmmmm . . . .

My current theory:
1. Jim had a good ticket from sometime when the App Pool had valid credentials.
2. During some subsequent tinkering, the App Pool credentials got hosed.
3. No new users were able to get tickets because the App Pool was unable to talk to the KDC. Jim however, was still able to access it with his cached good ticket (until the server reboot).
4. When we fixed the credentials of the App Pool, new connections got valid tickets.
5. Jim had some sort of bad ticket cached from his previous requests.

Anyway, good fun with SPNs, App Pools and Kerberos.

[1] KB Article 871179 -
http://support.microsoft.com/Default.aspx?id=871179

[2] ieHTTPHeaders -
http://www.blunck.info/iehttpheaders.html

[3] Troubleshooting Kerberos Errors -
http://www.microsoft.com/technet/prodtechnol/windowsserver2003/
technologies/security/tkerberr.mspx