Let’s Encrypt scrambles to manage fallout from digital certificate system bug

CAA check failure prompts recall of three million certs

UPDATE (March 9; 11:45 UTC) In a recent update to its community forum, Let’s Encrypt said it has been working with subscribers to replace more than 1.7 million blighted certificates.

Action remains pending for the remaining one million certificates affected by the bug. Rather than revoke these certificates once the deadline hit, Let’s Encrypt has decided that a stay of execution will cause less disruption.

Let’s Encrypt has warned that it needs to replace more than three million digital certificates that were compromised as the result of a Certificate Authority Authorization (CAA) bug.

Around 2.6% of the more than 116 million active certificates issued by Let’s Encrypt were affected by the problem.

The non-profit Certificate Authority (CA) has now started the task of renewing and replacing the affected certificates, where possible. Failing that, the certificates will be revoked.

The revocation process is due to begin today (March 4) at 20:00 UTC and needs to be completed before a Thursday, March 5 deadline.

Let’s Encrypt has already started contacting affected subscribers, and its advice is summarized in a community blog post that appeared soon after the bug was discovered.

The problem – disclosed over the weekend – seems to have been introduced in July 2019, hence the reason so many domains are affected.

In a notification email sent to domain holders, Let’s Encrypt warns that in cases where it is unable to renew certificates (and until freshly minted digital certificates are put in place), visitors will be confronted with an alert warning them that the site is insecure.

Developers have put together an online tool that allows web admins to determine if they have been affected, as well as publishing a list of all the affected certificate serial numbers.

What is CAA, and what went wrong?

CAA is a mechanism that creates a means for websites to specify which Certificate Authorities they authorize to issue certificates for their domain name.

Jacob Hoffman-Andrews, senior staff technologist at the Electronic Frontier Foundation, explained that the bug stemmed from how Boulder, the certificate management environment used by Let’s Encrypt, checked domain names.

“Our preliminary investigation suggests the bug was introduced on 2019-07-25 [late July],” Hoffman-Andrews added. “We will conduct a more detailed investigation and provide a post-mortem when it is complete.”

Mixed reactions

After identifying the bug in its CAA checking process, Let’s Encrypt suspended the creation of new certificates for just over two hours on Saturday in order to allow itself time to patch the problem.

“I’m not too concerned about the risks here,” web security expert Scott Helme said on Twitter.

“Reading the incident report, it seems the chances of something bad happening are exceptionally low. It can’t be stated for sure whether a mis-issuance actually took place.”

“Let’s Encrypt are handling this in exactly the way I’d expect them to, with 100% transparency and accountability. This isn’t a reason to stop using Let’s Encrypt. If anything, this is a reason to *start* using Let’s Encrypt,” he added.

Helme went on to blog about the issue.

UK-based infosec professional Kevin Beaumont was also impressed with the handling of the problem by Let’s Encrypt, even though he predicted the problem was likely to cause some disruption.

“Let’s Encrypt are revoking 3m certificates tonight,” Beaumont said on Twitter. “Really good transparency here, expect some websites and services to break but Let’s Encrypt renewal process is pretty polished.”

Some, however, expressed concerns that the issue could result in potential inconvenience and confusion.

Tim Mackey, principal security strategist at the Synopsys Cybersecurity Research Center, said: “Certificate revocation, while rare, does occur and web site owners should be prepared for this situation.

“Assuming that any certificate will remain valid until its complete expiration date is unrealistic. While it is inconvenient to perform an emergency update, processes should be in place within an organisation to handle such scenarios.”

What is Let’s Encrypt?

Let's Encrypt is a non-profit certificate authority set up to make encrypted connections to websites the norm.

The organization has issued more than a billion domain-validated certificates since its formation five years ago.

The free-of-charge certificates it issues are valid only for a maximum of 90 days and are offered alongside automated certificate renewal process.

READ MORE Let’s Encrypt deploys new domain validation technology to mitigate BGP hijacking risks