Exchange 2013 and SHA256 certificates

When using internal certificates for Exchange 2013, nowadays it’s common for customers to request SHA256 certificates as SHA1 might soon be vulnerable to brute-force attacks, given the ever
increasing compute power that is available. SHA256 is the same as SHA2, by the way. To read Microsoft’s opinion about SHA1
deprecation, see here.

Creating a SHA256 certificate for Exchange is fairly simple, as
explained here. When your certificate is ready, you assign it to all
services and reboot.

Now all of this may seem to work until you try to open the ECP or OWA. The login page opens up just fine but whatever credentials you enter, you only end up again where you started. Frustrating!

So what is wrong? No, you ECP virtual directory does not need to be recreated (as explained here). All Exchange services are running, the Event Log looks OK, and mail flow isn’t impacted. Recycling
application pools, running iisreset, or even rebooting the server does not help. And ECP and OWA authentication should not be touched unless you have very good reasons to do so. What then?

It turns out that Exchange 2013 isn’t compatible with SHA256
certificates as they are signed with the Microsoft Software Key Storage Provider. Instead, only the Microsoft RSA SChannel
Cryptographic Provider is supported – but that is not an option while creating your SHA256 certificate.

To verify this, run certutil -store my in a command prompt:

my “Personal”
================ Certificate 0 ================
Serial Number: 1D000000173f9d608d46c15e18000100000047
Issuer: CN=CA, DC=domain, DC=local
NotBefore: 14/12/2015 13:46
NotAfter: 14/12/2017 13:56
Subject: CN=*.domain.local
Non-root Certificate
Template: ExchangeWebServer, Exchange Web Server
Cert Hash(sha1): d2 fe e6 3f 23 87 74 d0 6b f6 b9 df b4 ee e4 63 12 5a a1 71
  Key Container = ld-ExchangeWebServer-b994773e-2f19-4d8f-ac58-ae168bd8d4f2
  Unique container name: b71b68251ae6117453ece6978affadcd_0caf5f43-47b0-49ad-b241-ce4a91571345
  Provider = Microsoft Software Key Storage Provider
Encryption test passed

 

If it sais “Provider = Microsoft Software Key Storage Provider” for your Exchange certificate, you have a problem.

So what are your options here?

  • Follow best practices and use a public certificate instead. While creating your certificate request, follow the SHA1 route. Then, when sending the request to the public CA, clearly indicate on their webpage you need a SHA256 certificate instead, even if your .req file sais you want an SHA1 one. When you receive the certificate, assign it using Enable-ExchangeCertificate
    -ThumbPrint “…” -Services “IIS,IMAP,POP,SMTP” and reboot your server.
  • If you want an internal certificate, this would mean that security is less of an issue. This may be a test setup, or perhaps the SSL tunnel ends at the load balancer? Anyway, an SHA1 certificate would do just fine.

Backup succeeded but some components backed up were inconsistent

One of my clients had issues running a full VSS backup of his Exchange server.  Windows Server Backup reported “Backup succeeded but some components backed up were inconsistent”. This is on a Windows 2012 R2 server with Exchange 2013 SP1 CU9.

Backup inconsistent

The event log showed several errors and warnings related to the failed backup:

Event ID 2227, source MSMQ: Backup failed during event OnPrepareSnapshot. Error 0x80070005: Access is denied.

MSMQ 2227

Event ID 16389, source SPP: Writer MSMQ Writer (MSMQ) experienced retryable error during shadow copy creation. Retrying…  More info: .

On the Details tab, some extra info can be found: “The writer’s timeout expired between the Freeze and Thaw events. (0x800423F2)”

SPP 16389

Event ID 8229, source VSS: A VSS writer has rejected an event with error 0x00000000, The operation completed successfully.
. Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer.

Operation: PrepareForSnapshot Event

Context:
Execution Context: Writer
Writer Class Id: {7e47b561-971a-46e6-96b9-696eeaa53b2a}
Writer Name: MSMQ Writer (MSMQ)
Writer Instance Name: MSMQ Writer (MSMQ)
Writer Instance ID: {123a679c-0d61-4f0c-bad4-74e39d6ecca0}
Command Line: C:\Windows\system32\mqsvc.exe
Process ID: 1356

VSS 8229

Troubleshooting steps that didn’t help:

  • When running vssadmin list writers, it shows you a list of all available VSS writers. Right after the failed backup, the MSMQ Writer will be in a failed state.
  • Reregistering the VSS writers, as explained on several sites, did not help, and neither did rebooting the server nor restarting the Message Queuing service.
  • Manually extending the VSS timeout, by increasing the value of HKLM\Software\Microsoft\Windows NT\CurrentVersion\SPP\CreateTimeout, had no effect.

The solution: in the log file of the failed backup, it also showed error 0x800423F2 (which means timed out) and a reference to several files in the C:\Windows\System32\msmq folder. MSMQ (the Message Queuing service) is one of the prerequisites of Exchange 2013 so it cannot be removed, and it’s also part of a full Exchange backup so it cannot be skipped.

Turns out that the msmq folder contained a subfolder called config. When browsing to that folder using a domain admin account, I got the message that administrative permissions were needed. Huh?

I took ownership of the folder and added full control for admins. From then on, the backup worked again. And the config subfolder? It was gone after the first successful backup. Go figure…