How Malicious Software Updates Endanger Everyone

You probably tell your users to update their software frequently, and that’s for a good and simple reason: Software updates fix vulnerabilities and make systems more secure. Not only that, they make the entire digital ecosystem more secure. A network is stronger when the links are strong. And conversely, a strong network can be hacked if there’s a weak link. For example, Target’s well-protected credit card network was compromised through weaknesses in its HVAC vendor’s computer system.

Notwithstanding the security risks, government agents may see malicious software updates as a means for surveillance. The term malicious software, or malware, refers to software code designed to be harmful to the software user, often by breaking down protective security measures and giving access or control to unintended third parties. The U.S. or other governments may seek to force you to create and install such malicious software. For example, an investigator could demand that you change your software to bypass passcode lockouts, enable wiretapping, turn on cameras, or physically track someone.

This risk is not theoretical. You may have read about Apple’s refusal to comply with a government order to create and push code to the San Bernardino shooter’s iPhone. In that case, the FBI asked Apple to develop software that would bypass an Apple security feature that erases the phone’s data after 10 incorrect attempts to enter the passcode. The government ultimately dropped its case when the FBI found a third-party hacker that had developed software that could break into the suspect’s iPhone. Apple has since claimed to make substantial security improvements in subsequent iOS updates, including to address the password-attempts loophole.

The likelihood that government actors may attempt to force software makers to push out software updates that include malware designed to obtain data from targeted devices grows as more companies secure their users’ data with encryption. And, as companies close other technological loopholes, there will be increased pressure on law enforcement to find alternate vulnerabilities to exploit. For example, Apple recently closed a technological loophole that law enforcement routinely relied upon to extract iPhone data. As other backdoors in secured devices are closed, the software update system will be an increasingly appealing option to law enforcement.

The risk of these practices raises serious privacy concerns for individual users. And malicious software updates also raise substantial cybersecurity concerns that could have serious effects on all of us, whether we use a particular product or not. Fixing vulnerabilities around the internet requires that the public trust the software update channel so that fixes to security weaknesses are deployed as soon as they’re made available. But people will not regularly update software if they fear the government or bad actors will use the new code to exploit their systems.

Governments may ask you for help, or they might seek to compel assistance. You have the right to say no to requests that are not backed up by a court order. By obtaining a court order demanding technical assistance, however, the government might attempt to compel you to install malware on a user’s machine as a software update that appears to be entirely ordinary, and that comes directly from you.

Law enforcement routinely seeks the assistance of software makers to obtain data in the course of criminal investigations. If you do not already hold the relevant data in a legible format, law enforcement might turn to “technical-assistance orders” that purport to require you to take novel and extraordinary steps to obtain unencrypted information about your users. These steps could include disclosing encryption keys, turning on microphones or cameras in consumer goods, circumventing password lock-out mechanisms, and more. One such novel form of technical assistance that might be attractive to law enforcement is the subversion of a software update mechanism. Such a technical assistance order might even include a secrecy clause purporting to prohibit you from telling anyone about the order.

You have the right to say no to requests that are not backed up by a court order. But by obtaining a court order demanding technical assistance, the government might try to compel you to install malware on a user’s machine as a software update that appears to be entirely ordinary, and that comes directly from you. You have a right to challenge these orders in court.

It is uncertain whether governments have already sought to obtain such orders or will do so in the future, but you should not wait to receive one before thinking through how to respond. We've compiled a guide with recommendations on how software developers and distributors like you can make informed decisions about protecting the integrity of your software update channels, both legally and technically. These pages seek to inform you on how to plan for and respond to such potential orders so that you are in a good position to resist, narrow, or defeat them — and protect your customers’ privacy and the security of the internet at large.

Here’s what you can do:

1. Understand the issue.
2. Implement privacy-minded designs and policies.
3. Plan responses to potential technical assistance orders ahead of time, rather than upon receipt.
4. Lawyer up!

If law enforcement were to ask you push a malicious software update to one or more of your users, it would likely do so by serving you with a court order for technical assistance similar to this one.

However, there are no clear legal obligations requiring you to do something as dangerous as subverting your software update channels to push malware, which can endanger your end users as well as other developers who may rely on your product.

Technical assistance orders may purport to be based on various legal provisions. If the underlying surveillance is based on a search warrant, the technical assistance order will likely be issued pursuant to the All Writs Act, a law that gives courts the authority to issue orders necessary to enforce other lawful orders, such as search warrants. Other sources of legal authority may be the Foreign Intelligence Surveillance Act, the Wiretap Act, or the Pen Register statute. Regardless of the legal authority for the technical assistance demand, the issues you will likely consider when deciding whether and how to comply will often be similar.

The pages on this website are not legal advice, but they are meant to help you think through how you might plan for and respond to a technical assistance order. If you receive a court order for technical assistance, you should talk to an attorney about your particular situation. There is a lot of uncertainty around the application of these laws, so your lawyer is in the best position to help you weigh your rights and options in deciding whether and how to comply with or challenge the order. You may consult your attorney even if the order comes with a non-disclosure provision.

Possible Scenarios

To help you plan, imagine situations in which you might receive a technical assistance order demanding subversion of a software update. Below are three potential scenarios that may arise under the All Writs Act:

1. Clever, a free web browser, receives a technical assistance order from federal law enforcement, which orders Clever to write an automatic update targeted at one user of the Clever browser. The investigators would like Clever to develop and incorporate malware into a software update sent to that user to enable them to monitor the user’s online activity to collect evidence of a crime.
2. Arthur is a release manager at Penguin, a free operating system popular for its security features and analytics tools. Arthur has received a technical assistance order from the FBI, and an accompanying gag order says he is bound to secrecy. The FBI has demanded that Arthur use Penguin’s automatic update network to push code to an individual it suspects of computer hacking. The code Arthur is told to push includes an FBI tool to log keystrokes; record web browsing history, usernames, and passwords; and list all the internet-facing ports open on the user’s computer. Arthur has reservations about complying with the order but is afraid to talk to anyone about it because of the accompanying gag.
3. TheGraham is a social media app that is widely used by people to share pictures of their food. TheGraham users are a dedicated bunch and post photos with great frequency throughout the day in an effort to share and discover their next best meal. Usually, the location where the photo was taken is shared along with the image. Law enforcement has sent a technical assistance order to TheGraham ordering that they send a non-targeted update to all users containing a patch that would enable law enforcement to trigger at-will location tracking for specific users. This request is not pursuant to any specific investigation; law enforcement has stated it will only use this ability pursuant to a warrant in the future.

Implement privacy-minded designs and policies.

As someone who develops or supplies software, you and your organization have an obligation to consider the risk that your software update channel presents to your users. As a responsible software distributor, you should take steps to defend your users against misuse of this channel. Similarly, users should prefer software distributors like you—ones that offer these defenses.

The First Line of Defense: Cryptographic Signatures

The most basic protection that every software distributor should offer is strong cryptographic signatures over the updates. As a software distributor, you should have a secret key, and your users should know about the corresponding public key. They protect their systems during the update process by verifying that the proposed update is signed by your public key. The security depends on your retaining exclusive control over the corresponding secret key so that malicious updates will not verify.

However, your secret key can be compromised (i.e. used to sign malicious updates) by legal compulsion, technical attack, or economic coercion. So more accountability is needed for a trustworthy update system than signatures alone.

Defense Against Targeted Attack

One possible attack on a user via software updates is the targeted distribution of malware. That is, an attacker who gains control of a distributor’s secret key can sign a malicious update and ship it only to the targeted user. Other users will continue to see the typical, non-malicious update channel.

Mirrorable Distribution

To bolster user defense against targeted attack, you should adopt “mirrorable distribution.” This means enabling users to pass information about available updates between one another, or allowing any number of intermediaries to copy and re-distribute (“mirror”) that information. If the attacker cannot predict where the user will pull their update from and cannot identify the user when they do an update, it is significantly harder to target a specific user. Letting a user fetch software updates anonymously from a range of sources concretely demonstrates that it is likely to be difficult to subject them to targeted attack.

In contrast, it is problematic to cryptographically sign software updates that are specific to a particular user or group of users. Targeted software updates facilitate targeted malware. For example, Apple’s iOS will not update unless it verifies a signature over the specific identifier for an individual phone or other device. This is dangerous both because it facilitates and incentivizes targeted malware and because it means Apple can “fingerprint” devices. You should ensure there is no tradeoff between end-user privacy and secure software updates.

Binary Transparency

An additional form of defense against targeted attack is an accountability mechanism known as binary transparency.

A mechanism protected by binary transparency will refuse to install an update unless the software has been verifiably logged in a global, irrevocable, auditable log. Website certificates — the mathematical objects behind the lock symbol in your web browser’s address bar — are already logged in a similar way. There are many ways to do this technically, but the tools are still relatively new and not widely tested. Mozilla is leading the way on this, but more work is needed, including tools for auditors to review binary transparency logs to highlight unusual behavior.

Auditability and Verifiability

If you take software update safety seriously, you will want to hold yourself accountable to your users. Accountability starts with auditability and verifiability. Even when individual users do not do their own audits or verification, the fact that others can do so provides a level of "herd immunity” — flawed updates can be detected and corrected even if your company is prevented from doing so.

Free/Libre and Open Source Software, also known as “FLOSS,” has the benefit of being readily auditable. FLOSS is a contract between the software distributor and the software user — it says that the human-readable source code that corresponds to the software update will be available to the user for inspection and improvement. With inspectable source code, it becomes much harder to hide malware inside software updates and patches. And with modifiable source, it becomes easier to fix problems that may have been introduced by the initial supplier. While some vendors will continue to sell proprietary software, visible and independently fixable code brings important security advantages.

You can also offer additional reassurance against malware with reproducible builds (a.k.a. deterministic compilation). Reproducible builds focus on how a developer translates software from code written by software engineers (source code) to machine code that computers can read and run (binaries). If the build is reproducible, the set of tools that do that translation (compiler, assembler, etc., sometimes referred to collectively as the “toolchain”) can not be modified to inject malware into those binaries. A compromised toolchain is not a theoretical attack.

If an uncompromised toolchain acting on the same source code produces a binary completely identical, down to the last bit, to all the other binary versions compiled by the vendor, then we can be confident that the vendor’s toolchain was not compromised. If not, it suggests that something may be, but is not necessarily, wrong. Toolchains sometimes produce minor differences between binaries because the compiled version includes trivial varying elements such as timestamps, tags indicating the name of the compiling computer, or more esoteric harmless changes. But serious security flaws can also be tiny, which makes them difficult to distinguish from inconsequential variation. With a reproducible toolchain, anyone who compiles the source code themselves can verify that it exactly matches a binary offered by a vendor (which is what the vast majority of software users are going to download).

Reproducible builds provide the most meaningful protection for users of Free/Libre and Open Source Software, because FLOSS vendors have already committed to having meaningful, visible source code that other people can review, modify and, double-check for themselves. But even if you are a vendor of completely proprietary software, you can take advantage of reproducible builds internally, as a way of ensuring that an order can’t be placed on one part of the organization (e.g., the build/release team) without other parts (e.g., the team that verifies reproducibility) knowing about it. If you have contractual agreements to provide copies of source code to auditors (even if you supply prorietary, non-FLOSS software), you can make reproducibility a part of their audits. And if you make a public commitment to reproducible builds, you will let your users know that a compromised toolchain can’t be used as an excuse for shipping malware.

Among other things, reproducible builds can protect individual software developers from extortion or personal attack because any malware slipped into a build through compromise of their build systems will be discovered as soon as someone else compiles the source code independently and discovers the mismatch. If you employ software developers, you can simultaneously reduce your organization’s attack surface, and reduce potential threats to your employees with this improvement.

Build reproducibility has in recent years become much more of a focus within the developer community, and non-reproducibility is increasingly considered a bug. Thanks to years of effort, the majority of widely used free and open source software packages, including the Apache webserver, the Linux kernel, and the Tor Browser are now reproducible, and work is ongoing across the ecosystem to establish this as a new norm in software development.

Providing Responsible Software Updates

Your users depend on you to provide them with updates that are concrete improvements and are not designed to harm them. While all software developers will make mistakes, you can provide your users with concrete defenses against potential malware and put in place systems that provide deterrence against attempts to use your update channels for stealthy injection of malware.

As a software user, if your preferred software distributor doesn’t apply the mechanisms and principles described above, consider encouraging them to adopt these approaches, or consider other software distributors that can offer you comparable functionality with better protections.

The technical protections above should of course be combined with legal and policy guidelines that make it clear that the software update channels should be inviolable as a critical piece of public infrastructure. But even if we get legal and policy constraints that keep some governments from trying to abuse these channels, less principled governments and non-governmental actors (organized crime, industrial espionage, etc.) may attempt to ship malware by these channels, so the technical guards described here are a necessary part of any healthy software ecosystem.

Plan for potential technical assistance orders ahead of time, rather than upon receipt.

Internal discussion among your executives, software designers and engineers, and lawyers is a helpful and important step to take prior to receiving a technical assistance order. If you are able to think through technical, legal, and business resources available to you in responding to such an order, you will be in a much better position to protect your software and your users should you receive one. You will also be more aware of how thoughtful, privacy-minded technical design can help protect your users by incorporating transparency and accountability in your software update channel.

Now that you understand the potential risks from government efforts to subvert your software update channels to distribute malware for surveillance purposes, here are some questions to consider to mitigate those risks.

Software Update Policies Generally

The policies and technology of your software update mechanism should be designed to guard against the risk of government ordered malware installation. Additionally, you may want to tell your users that you are taking steps to protect the software updates they depend on from compromise, including potential technical assistance orders. These honest representations will encourage your users to install updates, and may be relevant to whether you can raise First Amendment arguments against such orders later on.

1. Consider implementing technical measures to create more transparency in your software update channel, such as reproducible builds or binary transparency. See our review of technical defenses.
2. Consider how your software update channel operates. Should more than one person need to sign off on a software update before it can be sent out? Who should be in the loop?
3. What is your public commitment on the security of your software update channel? Does the internal process you have in place to address the possible receipt of a technical assistance order concerning your software update channel reflect that public commitment? Have you taken any steps to publicly communicate that when you cryptographically sign a software update or otherwise issue new software, you are confirming that it does not contain any malware, government-requested or otherwise? Have you articulated this position on your website or in the terms and conditions of your software?

Before You Receive a Technical Assistance Order

Make a plan for how to respond if you receive a technical assistance order using the following questions as a roadmap.

1. If you were to receive a technical assistance order, who would be the critical people on your team to inform?
2. Know that you may consult a lawyer even if the technical assistance order includes a nondisclosure order (also known as a “gag order”)—but depending on what such an order says, not all members of your team will be entitled to see, discuss, or respond to it at the time, and your lawyer will also be bound by the order. Do you have an employee who is designated to receive law enforcement requests? Do you have a general counsel, or other outside legal counsel already retained to handle law enforcement matters? Have you discussed with your counsel if they are familiar with various types of law enforcement requests, and whether they have ideas about potential ways to respond or resist such requests? If you do not presently have counsel, who could you reach out to in order to prepare for the event you received a technical assistance order?
3. Why might you resist a technical assistance order in a given case? How would complying with such an order affect the security of your software update channel? How would complying with such an order affect your users’ trust in your software? How would complying with such an order affect your resources, your project, or your business in the future should you have to comply with subsequent orders?
4. If you decide you would likely resist a technical assistance order that you believe is unlawful or dangerous, do you have the resources—financial and human—to resist such an order? Why or why not? If you feel you do not presently have the capacity to resist an order, what steps could you take to better position you to do so?

When You Receive a Technical Assistance Order

Contact a lawyer—and/or let the ACLU know.