Between October 21st and 25th, Smashing Magazine became completely unavailable for a majority of visitors. Visiting Smashing Magazine would give most returning visitors with a modern browser a security warning message like this:
Some other visitors would get a slightly different screen because of Smashing Magazine’s Service worker kicking in, and showing a placeholder “You’re Offline” message, but the underlying cause was the same: HTTP Public Key Pinning.
Smashing Magazine used to send an HTTP header with each response that looked like this:
Public-Key-Pins: pin-sha256="8RoC2kEF47SCVwX8Er+UBJ44pDfDZY6Ku5mm9bSXT3o="; pin-sha256="78j8kS82YGC1jbX4Qeavl9ps+ZCzb132wCvAY7AxTMw="; pin-sha256="GQGOWh/khWzFKzDO9wUVtRkHO7BJjPfzd0UVDhF+LxM="; max-age=31536000; includeSubDomains
Sites such as securityheaders.io encourage setting and at Netlify, we often get questions from clients on how to setup public key pinning. In theory, this header can be a strong weapon against the threat of “Man in the middle attacks.” These are attacks where someone would impersonate Smashing Magazine with a false certificate somehow generated via a Certificate Authority that your browser already trusts.
The key pinning header above tells browsers to refuse to accept any certificate that hasn’t been signed with one of the three keys indicated in the header for one year after visiting the site. Not just for
www.smashingmagazine.com, but also for all other subdomains.
When Key Pinning Goes Wrong
Key pinning protects against a relatively rare attack that’s very hard to pull off and that’s not a major threat scenario against a content-driven website like Smashing Magazine, but it does so at the cost of potentially causing major — in the worst case even catastrophic — outages.
For Smashing Magazine this happened when they were updating their expiring SSL certificate. They created a new wildcard certificate, added the digest of the new private key to the Public-Key-Pins header and rolled out the new certificate with a header looking like this:
Public-Key-Pins: pin-sha256="35L+K6PY5ynTu15SYPrT8KXp5TRH8kzP46mYLpv9k30="; pin-sha256="78j8kS82YGC1jbX4Qeavl9ps+ZCzb132wCvAY7AxTMw="; pin-sha256="GQGOWh/khWzFKzDO9wUVtRkHO7BJjPfzd0UVDhF+LxM="; max-age=31536000; includeSubDomains
Have you spotted the problem?
The old header had told visitors to Smashing Magazine that their browser should never accept any certificate that wasn’t listed in the old key pinning headers for the next 365 days. Adding the new certificate to the key pinning headers does nothing for this and the moment this went live, all previous visitors with a browser that had pinned the old certificate were now completely unable to visit Smashing Magazine!
Even worse, this could not simply be rolled back since the old certificate had expired!
Moral Of The Story
Consider really carefully if key pinning is worth it for your site. If you’re running a major bank or a secure site for whistleblowers, you’ll no doubt want to do anything you can to avoid possible man in the middle attacks.
However, if you’re running a content-driven website, it’s just never worth it to use a security technique that could potentially take your whole site down close to permanently! Going offline is a much more severe threat than getting exposed to a sophisticated man-in-the-middle attack.
If you really think you need public key pinning, consider at least setting a
max-age that’s low enough that you can survive it if something goes wrong. And do hold on to those precious pinned keys as if your life depended on it. Once you pin, there’s no going back if you make a mistake!
In a follow-up article, I’ll elaborate how to issue a new certificate that is using the keys of the old expired SSL certificate. Thanks for reading!