Menu Search
Jump to the content X X
Smashing Conf Barcelona

You know, we use ad-blockers as well. We gotta keep those servers running though. Did you know that we publish useful books and run friendly conferences — crafted for pros like yourself? E.g. our upcoming SmashingConf Barcelona, dedicated to smart front-end techniques and design patterns.

Be Afraid Of HTTP Public Key Pinning (HPKP)

Between October 21st and 25th, Smashing Magazine became completely unavailable for a majority of visitors. Visiting Smashing Magazine would give most returning visitors with a modern browser a security warning message like this:

A security warning message stating that your connection is not private1

The warning message most of Smashing Magazine’s visitors were seeing.

Some other visitors would get a slightly different screen because of Smashing Magazine’s Service worker2 kicking in, and showing a placeholder “You’re Offline” message, but the underlying cause was the same: HTTP Public Key Pinning.

Smashing Magazine used to send an HTTP header with each response that looked like this:

Public-Key-Pins: 
pin-sha256="8RoC2kEF47SCVwX8Er+UBJ44pDfDZY6Ku5mm9bSXT3o=";
pin-sha256="78j8kS82YGC1jbX4Qeavl9ps+ZCzb132wCvAY7AxTMw=";
pin-sha256="GQGOWh/khWzFKzDO9wUVtRkHO7BJjPfzd0UVDhF+LxM=";
max-age=31536000; includeSubDomains

Sites such as securityheaders.io3 encourage setting and at Netlify4, we often get questions from clients on how to setup public key pinning. In theory, this header can be a strong weapon against the threat of “Man in the middle attacks.” These are attacks where someone would impersonate Smashing Magazine with a false certificate somehow generated via a Certificate Authority that your browser already trusts.

The key pinning header above tells browsers to refuse to accept any certificate that hasn’t been signed with one of the three keys indicated in the header for one year after visiting the site. Not just for www.smashingmagazine.com, but also for all other subdomains.

When Key Pinning Goes Wrong

Key pinning protects against a relatively rare attack that’s very hard to pull off and that’s not a major threat scenario against a content-driven website like Smashing Magazine, but it does so at the cost of potentially causing major — in the worst case even catastrophic — outages.

For Smashing Magazine this happened when they were updating their expiring SSL certificate. They created a new wildcard certificate, added the digest of the new private key to the Public-Key-Pins header and rolled out the new certificate with a header looking like this:

Public-Key-Pins: 
pin-sha256="35L+K6PY5ynTu15SYPrT8KXp5TRH8kzP46mYLpv9k30=";
pin-sha256="78j8kS82YGC1jbX4Qeavl9ps+ZCzb132wCvAY7AxTMw=";
pin-sha256="GQGOWh/khWzFKzDO9wUVtRkHO7BJjPfzd0UVDhF+LxM=";
max-age=31536000; includeSubDomains

Have you spotted the problem?

The old header had told visitors to Smashing Magazine that their browser should never accept any certificate that wasn’t listed in the old key pinning headers for the next 365 days. Adding the new certificate to the key pinning headers does nothing for this and the moment this went live, all previous visitors with a browser that had pinned the old certificate were now completely unable to visit Smashing Magazine!

Even worse, this could not simply be rolled back since the old certificate had expired!

Moral Of The Story

Consider really carefully if key pinning is worth it for your site. If you’re running a major bank or a secure site for whistleblowers, you’ll no doubt want to do anything you can to avoid possible man in the middle attacks.

However, if you’re running a content-driven website, it’s just never worth it to use a security technique that could potentially take your whole site down close to permanently! Going offline is a much more severe threat than getting exposed to a sophisticated man-in-the-middle attack.

If you really think you need public key pinning, consider at least setting a max-age that’s low enough that you can survive it if something goes wrong. And do hold on to those precious pinned keys as if your life depended on it. Once you pin, there’s no going back if you make a mistake!

In a follow-up article5, I’ll elaborate how to issue a new certificate that is using the keys of the old expired SSL certificate. Thanks for reading!

(ms)

Footnotes

  1. 1 https://www.smashingmagazine.com/wp-content/uploads/2016/10/your-connection-is-not-private.png
  2. 2 https://www.smashingmagazine.com/2016/02/making-a-service-worker/
  3. 3 https://securityheaders.io/
  4. 4 https://www.netlify.com
  5. 5 https://www.smashingmagazine.com/how-to-issue-a-new-ssl-certificate-with-an-old-ssl-key/

↑ Back to top Tweet itShare on Facebook

Matt Biilmann has been building developer tools, content management systems and web infrastructure for more than a decade. He is co-founder and CEO of Netlify, the premium static hosting platform. In his spare time he drinks Jazz and listens to Beer, while helping to organize the SF Static Web-Tech Meetup.

  1. 1

    Hi Mathias, thank you for this article.
    what I fail to understand; what happens if I change my documents’ header to not set Public-Key-Pins anymore. Let’s say I tried and played around with HPKP, got it all to work and now decide that this simply isn’t the right choice (for example for reasons you outlined above). So I remove all corresponding headers.
    What happens then? Will the browser of a user tha visited before refuse to connect, and so never gets the info about the no-longer wanted keys? Are users forced to remove some deeply buried ssl caches somewhere?
    What happens if I in a first step make the HPKP caching period extremly small, like one hour, let it sit for a week and then remove all pinning headers?
    cheers,
    Tom

    1
    • 2

      Mathias Christensen

      October 26, 2016 2:32 pm

      I would recommend the latter approach of starting to serve HPKP headers with a low max-age and let it run for a while. Eventually you depend on people visiting your site while the new headers are in place, in order to pick them up.

      If you have a 1 year max-age in your HPKP headers today and someone visits your site and then comes back in 3 months, there’s nothing you can do if you’ve made any change to your certificate that’s not compatible with those old HPKP headers…

      -2
      • 3

        Wow. So even if later my site won’t send those headers, the browser will honor the pins/certs it has cached before?
        I guess this makes sense, otherwise a m-i-m simply needs to unset those headers and the whole thing is useless… but yeah, I think for a normal content based website this is a way too heavy tool, and needs very good consideration. The current encrypt everything trend here leads to serious implications for Joe Homewebsite… ^^

        0
        • 4

          Markus Seyfferth

          October 26, 2016 3:20 pm

          It does, but tomorrow we’ll follow up with a tutorial on how Mat managed to create a new SSL cert but with the old pinned key!

          1
    • 5

      You can deploy HPKP in report-only mode using the alternate header
      public-key-pins-report-only
      You can run this in production environments without having to worry about bricking your site in order to collect reporting data to see how HPKP would behave in the wild.

      4
      • 6

        Hey thanks – I knew I could just have a outage-friendly timeout, but report-only was all new to me!

        -1
  2. 7

    Great write-up. I’m a regular reader but didn’t experience a problem. How did Smashing Magazine mitigate the problem?

    0
  3. 9

    Another moral of the story is “don’t set max-age on HPKP to expire *after* the certificate it relates to” ;-)

    0
    • 10

      Mathias Christensen

      October 26, 2016 5:26 pm

      Well, the pinning generally doesn’t refer to the certificate, but to the private key used to sign the certificate, so the pinning can be longer than the certificate and there’s a small followup article on the way on how we solved the issue for Smashing.

      It doesn’t change that the worse case situation that people can get into with key pining is typically a way large risk than the kind of man in the middle attack this technique guards against…

      -1
      • 11

        This is incorrect. The pinning is a hash of the public key of either the certificate, the CA or the root.

        1
  4. 12

    Nick Sullivan

    October 26, 2016 6:15 pm

    Uhh… you can get a new certificate with the same key. Just resubmit the old CSR. HPKP is a key pin, not a certificate pin.

    2
  5. 13

    Actual date (time) browser will refresh pinned certs depends on maxAge which is number of days from user visit. So there is no way to tell, “hey browser cache this cert until 2 days before cert expires”. That’s IMHO not very clever design.

    -2
  6. 15

    Been trying to warn about this for a while (blogged about it here: https://www.tunetheweb.com/blog/dangerous-web-security-features/ ). As you state the attack vector is really quite rare and really, really believe that HPKP is overkill for most sites and there are workarounds that weaken it anyway (locally installed CAs like those used by virus scanners and superfish style software) cause HPKP to be ignored.

    I honestly think HTTP Headers are wrong place for this. DNS (where it could be updated without having to successfully connect to the site) seem a better place but that would require DNSSEC which has a lot of problems too.

    One final note is that browsers can (and I think do!) cap the max age of HPKP even if you set it longer in your header. For Chrome this is 60 days (https://bugs.chromium.org/p/chromium/issues/detail?id=523654) which is probably no bad thing!

    1
  7. 16

    After a few days of my phone only showing the Service Worker screen even though I was getting updates in my RSS reader, I assumed that something was bad in my browser cache. I cleared it and the site started working again.

    It’s great that you found a way to issue a new cert with the old keys. Waiting for every user’s cache to be cleared or expire would not have to been good for business!

    0
  8. 17

    Why is the article talking about Smashing Magazine in the third person? Is it written by Bob Dole?

    0
    • 18

      Mathias Christensen

      October 27, 2016 10:23 am

      I’m not a part of the Smashing Magazine team (I’m working on Netlify) – just happened to be around for an awesome Smashing Conf in Barcelona and helped them get these certificate issues fixed.

      1
    • 19

      Markus Seyfferth

      October 27, 2016 10:39 am

      The author Mat Biilmann Christensen works at Netlify, not Smashing, that’s why. :-)

      0
  9. 20

    From what I saw in the header now, I suppose the part of the fix is to put the old pin into the header again?

    0
  10. 21

    Marcus Bointon

    October 28, 2016 12:56 pm

    RFC 6844 (DNS Certification Authority Authorization (CAA) Resource Records) offers a simple DNS-based mechanism that has much in common with HPKP, but without the associated risk.

    As kevburnsjr mentioned, you can use the public-key-pins-report-only header which gets you many of the features of HPKP, but without the risk.

    One problem remains is that if you don’t set PKP, or set only PKP-RO, you are vulnerable to malicious pinning, where an attacker can set an invalid pin with a long expiry, thus blocking all your users. I think this is a major omission in the HPKP RFC, particularly in section 2.3.2, where PKP-RO does not override PKP.

    The best way to avoid this is to pin some common root certs such as Letsencrypt, Verisign or Comodo, giving you reasonable flexibility in obtaining new certs while reducing the security benefit only slightly. You should definitely set PKP-RO anyway along with reporting via Scott Helme’s great report-uri service.

    0
  11. 22

    hamza abdullah moh

    October 29, 2016 8:20 am

    This is really scary!!
    it did happen to me once when I issued a new certificate using cloudflare, but I was lucky because reversing back to the old one was an option.

    0
  12. 23

    I’m curious : which public key did pins #2 and #3 come from? I understand pin #1 was the old key of the server.
    As far as I understand HPKP, if The other pins came from the intermediate or root certificate and the new certificate had the same issuer, then the site should have remained accessible.

    1
  13. 24

    Visentin Davide

    December 24, 2016 2:13 pm

    That happened because you have not used correctly the key pinning mechanism. First of all, the max_age should be set so that the pin expire before the relative certificate expire. Moreover, you can set the so called backup pins to handle the case in wich you have to change a compromised certificate (the certificate fingerprint is calculated only on the subject field of the certificate, so it’s not necessary to have a backup certificate, but you simply need to forge a subject field with the public key that you want to use the next time).

    0

Leave a Comment

You may use simple HTML to add links or lists to your comment. Also, use <pre><code class="language-*">...</code></pre> to mark up code snippets. We support -js, -markup and -css for comments.

↑ Back to top