It sits in front of a website, routes requests, caches pages and stops one failure from spreading. Each replay below runs the same traffic twice, with one safety feature off and then on.
The traffic follows real Wikipedia page popularity. Partway through, something breaks: a cache node crashes, a page goes viral, or the server slows down.
One number per experiment, each the difference between the two recorded runs. Every value on this page is read out of the recorded data when the page loads.
These experiments can't run in a browser tab: they need real processes crashing, real sockets and real signals. So the page plays back recordings. The code that produced them is on GitHub, where it builds, passes its tests on every push, and runs in front of ordinary HTTP servers.
Cache-Control: max-age, passes everything else straight through, and never shares a cached response between requests that carry cookies or an Authorization header.examples/front-a-python-server.sh fronts a ten-line Python server, kills a cache node, and restarts the edge during 400 requests. All 400 return 200.lab run re-records every number on this page in about seven minutes.Every run happened on one laptop, so there is no real network latency or packet loss and the throughput is laptop-scale. The numbers show how much each mechanism changes the outcome, not how an internet-facing deployment would perform.
Three origins with sixteen workers each, a bounded queue and lognormal service times, about 2,300 requests a second in total. Like most real backends it keeps working on requests their callers have abandoned, which is what makes the retry storm self-sustaining. Its parameters are in the repo.
While the origin was four times slower, both sides served very little: 3% without the breakers and 7% with them. The difference is what happened afterwards. One side recovered half a second after the origin did. The other never recovered.
Consistent hashing, request coalescing, retry budgets and socket handoff all come from how existing CDNs and proxies work. What is mine is the from-scratch implementation, the experiments, and two bugs I found while measuring it. Nobody should deploy this instead of Envoy.