Cache bail

How a small CDN broke the Internet

I’m a researcher at the UC Berkeley Center for Long-Term Cybersecurity, where I direct the Daylight Lab. This is a newsletter about cybersecurity and politics—my work as I do it; more than half-baked, less than peer-reviewed. To follow along, subscribe here.

Sometimes, you write something at the exact right time. Last Friday, I wrote about centralization among caching services, and why that centralization poses a major threat to the Internet.

Four days later, Amazon went down. CNN went down. GitHub, Reddit, Twitch, StackOverflow were all down. What happened? A coordinated cyberattack?

Nah, just human error at Fastly, the second-largest caching service.

How did a cache break the Internet?

According to W3Techs, Fastly only has about 5% of the CDN market share. How was an outage in a relatively small CDN able to take down such a wide variety of important websites?

The answer is two-fold. First, CDNs sit at a critical point between user queries (give me nytimes.com) and server responses (here’s the New York Times). This position makes them the broker for Internet traffic. They’re the bouncers of the Internet. Everything goes through them.

Second, CDNs’ failures tend to cascade. Fastly may not have a large share of the market, but the things it does control are essential to a wide variety of people. When they go down, the issue percolates through supply chains, taking out downstream services like falling dominoes. The last dominoes in the chainx are often big names like Amazon.

It could have been worse

We’re lucky this incident was nothing more than an “oops” and a quick fix.

  1. It could have been Cloudflare. Cloudflare owns 80% of the CDN market. If Cloudflare went down, the chaos would be global. Imagine no one’s credit card working. That level of chaos.

  2. It could have been an attack. This was a misconfiguration. It could have been a government-sponsored attack. Not a bad thing to hit!

  3. The US government can compel CDNs to do things. As I wrote last week, almost all of these CDN services are based in the US. That gives the US leverage, through its court system, to demand CDNs take particular actions.

An American Great Firewall

The concentration of CDNs in the US could be the beginnings of an American Great Firewall. Here are some things the US could (in theory) compel CDNs to do.

  • “Block all traffic going to China.” That would cut China off from the global Internet.

  • “Disallow all traffic to this list of IP addresses.” That would be a “digital do not fly list.

  • “Don’t do business with Gab.” That would leave Gab exposed to the global Internet, easy for activists to take down via DDoS attacks.

Now what?

CDNs are critical infrastructure, just like electrical plants and water treatment facilities. Unlike that infrastructure, a tiny handful of companies run CDNs for the entire world. That centralization isn’t just a monopoly problem—it provides a central point of failure for the whole Internet.

  1. Antitrust action. Someone like Lina Khan should be looking at this market for possible antitrust action. It needs to be broken up. It’s a cybersecurity imperative as well as an imperative for allowing competition.

  2. Build decentralized alternatives. Open-source developers should try to create a decentralized, worker-owned CDN. This project may not be as flashy as decentralized finance (whatever that means), but it could be extremely impactful in helping to (re)decentralize basic Internet infrastructure.

  3. State-owned CDN? On the other side of decentralization, there’s an argument for a CDN run by a nation-state. This is a public utility that provisions a common good. A centralized system could be easier to protect and more agile in the face of new threats. A well-regulated public utility is an antidote to the dystopia of a US-led global censorship network.

Credit to my partner for the brilliant title. Check out Clare Duffy’s excellent article in CNN.

To follow my newsletter, subscribe here.