Creating standards for CDNs

Content delivery networks (CDNs) have been around for a long time, but they’re not all built in the same way. While Fastly’s edge cloud platform goes beyond traditional content delivery networks by moving things to the edge, there are more fundamental differences; it’s not uncommon for CDNs to make up their own rules about how they serve web traffic, since CDNs didn’t exist when HTTP was defined. To improve this, we’re working alongside other platforms to standardize basic protocol handling for CDNs.

A while back, security researchers noticed what had concerned engineers working on content delivery networks CDNs for some time; it’s possible to point competing CDNs at each other to bring them down. From the abstract of Forwarding-Loop Attacks in Content Delivery Networks:

Malicious customers can attack the availability of Content Delivery Networks (CDNs) by creating forwarding loops inside one CDN or across multiple CDNs. Such forwarding loops cause one request to be processed repeatedly or even indefinitely, resulting in undesired resource consumption and potential Denial-of-Service attacks. To evaluate the practicality of such forwarding-loop attacks, we examined 16 popular CDN providers and found all of them are vulnerable to some form of such attacks.

Because of the massive scale of many CDNs — with terabits per second of link capacity available to each, worldwide — this is a potentially very scary problem. Whether it’s an intentional attack or an accidental misconfiguration, it could take large parts of the internet down with the CDNs in question. In fact, one CDN engineer I was discussing this with recently said it was one of the problems that “kept him awake at night.”

Many CDNs, and Fastly’s edge cloud platform, already have loop protection mechanisms in place, usually by using a header to identify requests that have already been seen. The problem is that these solutions aren’t coordinated between each other, so that one CDN might be configured to delete another CDN’s loop detection headers — either intentionally or accidentally.

We wanted something better, so we started talking to our colleagues at other CDNs as well as content platforms. The result was a small specification in the HTTP Working Group for the CDN-Loop request header, which was approved for publication as a standards-track RFC by the Internet Engineering Steering Group (IESG) two weeks ago.

This is a very simple mechanism. For example, Fastly’s looks like this:

GET /index.html HTTP/1.1
CDN-Loop: Fastly

Implementing CDNs are required to add it on each request they make, and to protect it from accidental (or not-so-accidental) modifications, so that they can detect and mitigate such loops more reliably — even when a loop involves multiple CDNs.

We’re very happy about this small step forward because it’s a sign of change; since their early beginnings, there hasn’t been much coordination between CDNs about issues like this, much less in providing a consistent experience to our customers. So, while small, it’s an important step, as it signifies an industry that’s willing to work together.

We’re also not done; Fastly is contributing to a number of other community efforts to build better, more consistent experiences across vendors, including:

  • The HTTP Working Group has started work on the Cache response header to create a more consistent debugging experience for cache behaviour, both in CDNs and other HTTP caches.

  • We’re also contributing to a new, so-far unadopted draft on the Proxy-Status response header, to make CDN and reverse proxy debugging easier when something goes wrong.

  • The core HTTP specifications are being revised (again). This time, we’re paying special attention to how CDNs behave to make sure that they don’t need to violate the specifications on a regular basis.

  • That core work is being informed by an emerging set of common test cases for both browser and intermediary HTTP caches.

  • We’re looking at more advanced features like Variants to give intermediaries (including CDNs) more information about the content they’re serving, so they can make more intelligent caching decisions.

Once we get these and a few other basic building blocks for CDN behaviour standardised, we hope to move on to more exciting things — like standardising how an origin can discover that it’s behind a CDN, send purge requests to it, and even use surrogate keys without lots of configuration.

We believe that having one standard way to interact with a CDN makes our customers’ lives easier. It also makes it easier for content platforms like WordPress and Drupal to seamlessly work with CDNs when they’re in front of such a site — making things better for everybody.

Mark Nottingham
Senior Principal Engineer

4 min read

Want to continue the conversation?
Schedule time with an expert
Share this post
Mark Nottingham
Senior Principal Engineer

Mark Nottingham has helped to define and develop the web and the internet since the late 90s. He's written, edited, or substantially contributed to more than 30 IETF RFCs and W3C recommendations about topics like HTTP, caching, linking, web architecture, privacy, and security.

As chair of the the HTTP Working Group since 2007, he has overseen the evolution of the foundational protocol of the web, notably including HTTP/2. As chair of the QUIC Working Group, he oversaw the creation of HTTP/3 and the evolution of internet transport. He has also served in internet governance bodies, including the Internet Architecture Board and the W3C Technical Architecture Group.

Currently, he’s part of the Office of the CTO at Fastly, and studying Communications Law at Melbourne Law School. Mark is married to Anitra with two sons, Charlie and Bennet. They live in Melbourne, Australia.