Security Week 20: disabling Firefox extensions

On May 4, at four in the morning in Moscow (or a little later, depending on luck), all installed extensions ceased to work for users of the Firefox browser, and installing new add-ons became impossible. The problem was on the browser side - the intermediate certificate, by which all extensions are signed, has expired. This event has an indirect relation to information security - the bad luck arose as a result of the logical desire of developers to protect users from malicious extensions (starting in 2015) and due to the fact that no one noticed that the certificate will expire soon.

Nevertheless, this is an interesting story with a moderately happy ending: a fairly non-trivial problem was solved within 12 hours. This is also a well-documented incident, with a lot of information from both developers and incredible amounts of drama from users. In the process, the problem of privacy arose, which was quite effectively resolved.

The first reports of problems with extensions appeared before the certificate expired, May 3. There is at least one such discussion on Reddit. The author of the thread on the computer had the wrong date set, so he was one of the first to find out about the bug (and “fixed” it by setting the correct date, but not for long). A little later, the problem appeared for all users, but at different times: the validity of the certificates by which the extensions are signed is checked once every 24 hours. Further history with varying degrees of detail is described here: news , bug report , technical support , a blog post with technical details, and a story about the incident on behalf of CTO Firefox.


This is how the process of signing extensions in Firefox looks like. The root certificate at the top of the chain is in offline storage, and once every several years with its help an intermediate certificate is created with which the extensions are signed. It was the interim certificate that expired on May 4. The first workaround on the part of Firefox developers was the release of a patch that temporarily stops the validation of extension certificates. If such a patch flew to you before the check, then the problem has passed you. Further, the developers had two ways: to issue either a new release of Firefox, or a new certificate, which will make valid extensions signatures in the current version. It was impossible to re-sign all additions (more than 15 thousand). More precisely, perhaps, but it would have taken a very long time.


Given the tight deadlines (they found out about the problem in Firefox on the evening of May 3, of course, on Friday!), It was decided to investigate both options. Technically, there was an opportunity to issue a new certificate, a) it was necessary to generate this certificate and b) deliver it to users as quickly as possible. The first part was complicated by the fact that the root certificate is stored in a hardware module, which still needs to be reached ( in the forest? In the bank cell?) In addition, when the developers got to the root certificate, it was not immediately possible to generate a new intermediate certificate, and this each time led to the loss of an hour or two for the necessary tests. How to deliver? To do this, we used the Firefox Studies mechanism - in fact, the add-ons distribution system “from the browser developer”, which in normal cases is intended for experimental code. It turned out faster than building a new build and sending the update via regular channels.


But here the same privacy problem arose. Firefox Studies, as an experimental system, is only included when sending browser usage information back to Firefox developers. This is logical for beta testing, but it looks a little strange in the context of delivering a patch that all users generally need. This problem was elegantly resolved: Firefox decided to remove all telemetry received from May 4 to May 11.

This was not an ideal solution. Those with telemetry disabled (and Studies) had to manually enable these options. In some builds this option is not at all. The Android version of the browser does not support Studies. Permanently affected users of older versions of Firefox who do not want to be updated, but use add-ons. May 8 Firefox releases released66.0.5 and Firefox ESR 60.6.3 , in which the certificate problem is finally resolved, and Studies with telemetry no longer need to be included. Updates are planned for older versions of the browser, starting with Firefox 52. The problem was resolved, but for many users it did not pass without a trace - there are cases of data loss and settings in the extensions.

There are two conclusions from this story. Firstly, Firefox promised to set up a “time bomb” tracking system in the infrastructure to prevent this from happening in the future. Secondly, it became clear that even a more or less modern update distribution system, with different methods for delivering patches to users, is not as effective as we would like. According to CTO Firefox, browser users should be able to receive updates and hotfixes, even if they wish to disable any other experimental features and / or telemetry. This is a happy ending story, which may lead to an improvement in the update mechanism of Firefox, a rare mainstream browser that is not tied to any major IT company. But do not forget that she began with allegorical shooting on her own legs.

Disclaimer: The opinions expressed in this digest may not always coincide with the official position of Kaspersky Lab. Dear editors generally recommend treating any opinions with healthy skepticism.