Creating standards for CDNs
Content delivery networks (CDNs) have been around for a long time, but they’re not all built in the same way. While Fastly’s edge cloud platform goes beyond traditional content delivery networks by moving things to the edge, there are more fundamental differences; it’s not uncommon for CDNs to make up their own rules about how they serve web traffic, since CDNs didn’t exist when HTTP was defined. To improve this, we’re working alongside other platforms to standardize basic protocol handling for CDNs.
A while back, security researchers noticed what had concerned engineers working on content delivery networks CDNs for some time; it’s possible to point competing CDNs at each other to bring them down. From the abstract of Forwarding-Loop Attacks in Content Delivery Networks:
Malicious customers can attack the availability of Content Delivery Networks (CDNs) by creating forwarding loops inside one CDN or across multiple CDNs. Such forwarding loops cause one request to be processed repeatedly or even indefinitely, resulting in undesired resource consumption and potential Denial-of-Service attacks. To evaluate the practicality of such forwarding-loop attacks, we examined 16 popular CDN providers and found all of them are vulnerable to some form of such attacks.
Because of the massive scale of many CDNs — with terabits per second of link capacity available to each, worldwide — this is a potentially very scary problem. Whether it’s an intentional attack or an accidental misconfiguration, it could take large parts of the internet down with the CDNs in question. In fact, one CDN engineer I was discussing this with recently said it was one of the problems that “kept him awake at night.”
Many CDNs, and Fastly’s edge cloud platform, already have loop protection mechanisms in place, usually by using a header to identify requests that have already been seen. The problem is that these solutions aren’t coordinated between each other, so that one CDN might be configured to delete another CDN’s loop detection headers — either intentionally or accidentally.
We wanted something better, so we started talking to our colleagues at other CDNs as well as content platforms. The result was a small specification in the HTTP Working Group for the CDN-Loop request header, which was approved for publication as a standards-track RFC by the Internet Engineering Steering Group (IESG) two weeks ago.
This is a very simple mechanism. For example, Fastly’s looks like this:
GET /index.html HTTP/1.1
Implementing CDNs are required to add it on each request they make, and to protect it from accidental (or not-so-accidental) modifications, so that they can detect and mitigate such loops more reliably — even when a loop involves multiple CDNs.
We’re very happy about this small step forward because it’s a sign of change; since their early beginnings, there hasn’t been much coordination between CDNs about issues like this, much less in providing a consistent experience to our customers. So, while small, it’s an important step, as it signifies an industry that’s willing to work together.
We’re also not done; Fastly is contributing to a number of other community efforts to build better, more consistent experiences across vendors, including:
The HTTP Working Group has started work on the Cache response header to create a more consistent debugging experience for cache behaviour, both in CDNs and other HTTP caches.
We’re also contributing to a new, so-far unadopted draft on the Proxy-Status response header, to make CDN and reverse proxy debugging easier when something goes wrong.
The core HTTP specifications are being revised (again). This time, we’re paying special attention to how CDNs behave to make sure that they don’t need to violate the specifications on a regular basis.
That core work is being informed by an emerging set of common test cases for both browser and intermediary HTTP caches.
We’re looking at more advanced features like Variants to give intermediaries (including CDNs) more information about the content they’re serving, so they can make more intelligent caching decisions.
Once we get these and a few other basic building blocks for CDN behaviour standardised, we hope to move on to more exciting things — like standardising how an origin can discover that it’s behind a CDN, send purge requests to it, and even use surrogate keys without lots of configuration.
We believe that having one standard way to interact with a CDN makes our customers’ lives easier. It also makes it easier for content platforms like WordPress and Drupal to seamlessly work with CDNs when they’re in front of such a site — making things better for everybody.