This past week I've been working on some Silverlight controls that need to be hosted in and deal with users and groups in SharePoint, as well as their own tables in a SQL database.

This has meant hosting a WCF service below the SharePoint app so it can access some SharePoint context objects, and that has been an adventure.  I will save you the gory details and just link you to Sahil Malak's outstanding series of posts on SharePoint as a WCF Host. (with the following extra advice:  Read all the parts 1 through 4, but don't start implementing any of it until you read the Real World part 1 post in which he links to a wsp that does most of the annoying heavy lifting for you.  Don't waste your time implementing manually, just read the other posts so you understand what's happening.)

All that having been said and done, I still had no end of trouble making the endpoint work, due to the dreaded security configuration... 

One thing you need to know about WCF is that it's pretty pragmatic about security: it's not going to let you send a username and password in plain text. (Well, unless you go through the trouble of adding and parsing and validating it manually, but then you're just being a brat.  Take the framework's implicit advice and don't ever send credentials in plain text!)  Basically, any combination of mode, transport, and message settings in the security node of your binding declaration that would lead to plain text credentials will lead to one InvalidOperationException or NotSupportedException or another. 

Since we're consuming the WCF service using Silverlight, we need to use basicHttpBinding.  By default that uses and requires anonymous access.  We're in SharePoint, and our authentication methods include integrated Windows authentication and Forms authentication.  Anonymous access to this site is not allowed, and we don't want to enable it.  So our first attempt yields the following exception:

Security settings for this service require 'Anonymous' Authentication but it is not enabled for the IIS application that hosts this service.

Okay, that's reasonable.  So, as a first attempt, we could try and secure it by using the Forms authentication. (In this case we actually don't want the Forms users to have access, but we'll explore that option.)  So let's set the security mode to message and give it a message element with clientCredentialType="UserName"...

BasicHttp binding requires that BasicHttpBinding.Security.Message.ClientCredentialType be equivalent to the BasicHttpMessageCredentialType.Certificate credential type for secure messages. Select Transport or TransportWithMessageCredential security for UserName credentials.

Oops.  Right.  That would lead to plain-text passwords.  Reading that exception message, it's actually one of the more helpful ones I've seen.  It makes it clear to us that we have the choice of changing the clientCredentialType to "Certificate" (which we can't do because technically that's still anonymous access as far as IIS is concerned and we don't want to turn that on), or changing the security mode to Transport or TransportWithMessageCredential.  What does that mean, you ask?  Well let's try it:

Could not find a base address that matches scheme https for the endpoint with binding BasicHttpBinding. Registered base address schemes are [http].

When those say "Transport", they're referring to promising to WCF that it'll be called through SSL.  So if SSL is not an option for you, this doesn't help you.  Also, remember that all WCF cares about at this point is whether or not you're passing credentials in plain text, but once you get past this point by using https you'll be back to the part where you have to authenticate in a manner that satisfies IIS anyway.

"But wait!" you say, "this endpoint is only accessible from the internal network anyway, and integrated Windows auth over http is fine in this case!"  Well, fortunately you're not out of luck.  WCF will allow this scenario; the magic word is a security mode of "TransportCredentialOnly". Now you give it a transport element with clientCredentialType="Windows" and bam, you're using integrated Windows authentication!

...Except here's where I got stuck.

Doing that didn't make my endpoint start working.  Instead, it gave me this fun message:

 Security settings for this service require Windows Authentication but it is not enabled for the IIS application that hosts this service.

 Wait, what?  Of course Windows Authentication is enabled!  It's a SharePoint application hosted in IIS which is USING integrated Windows auth for sign-in. Visiting IIS Manager, we can look and confirm that Integrated Windows authentication is checked.

Here's the thing you need to know about Windows Authentication, if you weren't already aware:  there are two kinds of it.  Windows 2000 and later make use of Kerberos tickets for Windows Auth.  This is a nice and relatively secure option.  Previous to Kerberos being adopted by Microsoft, however, Windows Auth was accomplished using a Microsoft protocol called NTLM, which is less secure, but is OK and still in wide use.  Kerberos is preferred when available, but when it's impossible (you're not on the domain, or there's no Active Directory, or ports Kerberos needs aren't open, or any number of scenarios) then NTLM is automatically used as a fallback.

In the metabase for a site hosted in IIS 5.1 and later, there's a property called NTAuthenticationProviders.  This is a comma-separated list with a couple possible values, and it controls the behavior of Windows Authentication on that IIS site.  The value "NTLM" enables NTLM to be used, obviously.  The value "Negotiate" indicates that Kerberos should be used, unless while negotiating with IIS it's determined that Kerberos isn't possible, in which case NTLM will be used instead.

The default for this value is "Negotiate,NTLM".

For some reason, on my development VM, this was set to just "NTLM".

When you say clientCredentialType="Windows" in your endpoint's binding configuration, you're actually requiring the Negotiate option.  If only NTLM is available, you'll get the strange exception message about Windows Authentication not being enabled even though strictly speaking it is enabled.

I could probably have solved this problem by changing the metabase value back, but I hate touching the metabase and I don't need Kerberos level security for this scenario anyway.  The solution was thus to put clientCredentialType="Ntlm".  This got me up and running... finally.