I recently had to replace the public-facing service communication certificates on our primary ADFS deployment on Server 2012 R2. I followed a procedure that I thought had a reasonable chance of actually doing what I wanted it to:
- Obtained a new private key with signed certificate.
- Saved the file to a pfx, and imported it onto each node in the ADFS cluster
- Set permissions on the certificate according to documentation
- Used the ADFS MMC -> Certificates -> Set Service Communications Certificate.
Everything seemed to go okay, but after a bit we started to get some complaints that some of our users could not access the Office 365 Pro Plus software download page. This was a curiosity to me, because I could not reproduce the problem. A colleague later noticed a raft of SSL errors in the System event log on one of the ADFS nodes, and disabled it in the load balancer configuration.
When I finally got around to investigating, I noticed that the system log reported problems from source ‘HTTPEvent’, with details DeviceObject: DeviceHttpReqQueue, Endpoint: adfs.uvm.edu:443 (and also Endpoint: 0.0.0.0:443). What gives?
I found a related TechNet Blog that shed some light on the subject:
According to this document, after setting the Service Communications Certificate in the MMC, you must run:
to fetch the certificate thumbprint of the Service Communications Cert. Take note of the certificate thumbprint, then run:
Set-ADFSSslCertificate -Thumbprint [yourThumbprint]
“Set-AdfsSslCertificate” will fix the HTTP.SYS bindings used by ADFS. Apparently the MMC does not set the bindings, which is pretty annoying because this leaves the service in a pretty darn broken state. The HTTP bindings are mentioned in this TechNet documentation:
BUT, the docs do not explicitly state that the Set-AdfsSslCertificate cmdlet needs to be run on all of the ADFS server nodes in your farm. This also is a key missing detail.
Good Documentation… you always take it for granted, until you don’t have it anymore.
Note above that I mentioned a binding problem with the address 0.0.0.0:443. This was a carry-over from our initial deployment of ADFS 3. Back then, Microsoft did not provide a health check URL for ADFS, and the supplemental binding was needed to allow health monitor connections from our F5 load balancer without using SNI, which is required by ADFS 3.0, but not supported on the F5. These days (and if you have KB2975719 installed), you can instead monitor the following URL from your F5:
More details on this solution can be found here:
While it is nice having a proper health check, problems can arise when your ADFS server HTTP bindings go sour. It would seem that nothing is perfect.