A benchmark hands you a number. I spend most of my time on the thing that checks whether the number means anything: a simulator you can re-run yourself, an exact answer to compare it against, and tests that fail when a claim stops being true.
Each one re-runs its experiment right here in the page, on the same data the original study used, next to the original implementation. Nothing on this page is a recording of a result I got earlier.
Two identical fleets work through a year of real Seattle medic calls on the same clock. When the page loads, the browser engine re-runs the whole year and compares itself to the Python version I ported it from. Right now they disagree by 0.05 seconds.
OPEN ITType in any puzzle and the page counts all 108 ways to combine the numbers, so the difficulty is known exactly rather than estimated. Then two optimisers train on the same reward and you can watch one of them hit a gradient of exactly zero and stop being able to move.
OPEN ITA model writes up a Supreme Court argument, and every sentence it produces gets traced back to who said it and when, down to the second of audio. Plant a specific error yourself and see whether the detector catches it.
OPEN ITAn RL attacker hunts for ways to make a sandboxed tool-using agent break rules that are checked in code rather than judged by another model. Every time it finds one, I upgrade the defense and it starts hunting again.
OPEN ITA marketplace where a provider can only see you if they hold a licence in your state, a contract with your insurer and an opening before you give up. Taking the first slot that fits is what most systems do, and it leaves people unseen. Press run and watch the solver take bookings apart and put them back together.
OPEN ITFifty-six cards, each carrying a phrase about something a person did. You are told to ignore the words and drag by colour. Three minutes later it tells you how much slower you were at pulling the bad ones toward yourself. Built from a Duke lab's public preregistrations, and it shows you its own reliability rather than hiding it. This one looks different on purpose.
TAKE ITNobody gets compromised by an obviously malicious package. They get compromised by a patch release of something they already trust. Both versions of an install script actually execute in the page, and the diff shows which powers the new one picked up. The right-hand script is editable, so you can try to sneak something past it.
RUN THE DIFFA scanner tells you a string looks like a key. It never tells you that the key publishes to four million installs a week, or that IAM would let it make itself administrator. Move a credential between a keychain, a committed repo and a synced folder and watch the blast radius and the rotation order recompute underneath you.
REARRANGE ITA two-mile highway loop with cars that follow a real car-following model and four pavement sensors you can drag around. Drop cars, stall one between two sensors, and a small neural network trained live in a background thread, with vehicle conservation built into its loss, works out where the jam is in the blind spots. The same model was then run 187 times on real freeway data; the numbers, including where it loses, are on the page.
MAKE A JAMThe nine above run inside this page. These seven can't. The first is six deployed services and two Postgres databases, so instead of a demo I'll just give you a login and let you settle a real match on it. The second only proves anything when real processes crash, flood and get redeployed on real sockets, so you get the recorded runs instead, every 100 milliseconds of them. The third needs Quantinuum's quantum emulator, which is a Rust program, so it gets the same treatment. The fourth is a replicated control plane whose real numbers only exist when processes crash on real sockets, so its page replays a recorded run and also runs the same Go code in your tab as WebAssembly. The fifth is an image index built by a Node pipeline around a 150 MB vision model, so the page ships the finished index and only pulls the model into your tab when you type your own search. The sixth is 47 tumor-segmentation models trained over two and a half hours of CPU time, so its page shows their real predictions instead of training anything in your tab. The seventh needs a local language model and PyTorch, so you get a recorded session plus every benchmark output to browse.
A charity beach volleyball app, plus the competition platform underneath it. Both teams submit the score from their own phones and the system compares hashes instead of trusting either side. Payouts come out of a pure function, and a human has to close the contest behind a frozen preview before any of it moves. The ledger is append-only because the database role it runs as genuinely cannot update or delete a row.
OPEN THE WRITE-UPAn edge gateway: a proxy, five cache nodes and three origins, with the parts a CDN leans on. Consistent hashing, request coalescing, retry budgets, circuit breakers, load shedding and restarts that hand the open socket to the new process. Every experiment runs twice on identical traffic with one mechanism switched off, so you can watch a cache node die, a page go viral and an origin get buried by retries nobody is waiting for, side by side. The page replays recorded runs. The gateway itself is on GitHub, and it runs on Linux and macOS in front of any plain-HTTP server.
WATCH THE REPLAYSA CI gate for quantum programs. Their right answer is random, so each test checks the shape of thousands of outputs with statistics that cap both flaky failures and missed bugs at 1 in 1,000, and stops spending shots once the answer is clear. Three of the eleven bugs I planted pass every normal measurement every time, and one extra check in a different basis catches each of them in 250 shots.
WATCH A BUG GET CAUGHTA deployment control plane for cloud regions that drop off the network, or never had one. Three replicas keep one Raft log of what every region should run. Connected regions pull from it, and an air-gapped one only accepts signed packages carried in by hand. Kill the leader, cut a region off, crash a worker between installing and reporting back, or tamper with a package in transit, and every region still ends up on the right version exactly once. The page replays a recorded run and runs the Go code in your tab, compiled to WebAssembly. The failover times come from real processes on real sockets.
TRY TO BREAK ITVisual search the way Pinterest does it, small enough to inspect. Every image and query goes through a real CLIP model, then a graph of visual neighbors and board-mates re-ranks the results with a personalized PageRank walk. Flip one switch to compare against CLIP alone: for a brand-new pin with its board hidden, the graph puts 84% of its board-mates in the top ten, against 70% for CLIP. Drag a box around any object to search with just that region.
COMPARE THE RANKINGSA generator that paints realistic fake tumors into real mouse micro-CT scans, so a tumor-finding model can learn from a handful of hand-outlined ones. With two labeled scans, adding the fakes did about as well as six real ones and cut tumor-volume error from 75% to 35%. Simple spheres are the control: they help a little at first, then make the model worse once there are eight labeled scans. By then the fakes stop helping too, so it's a cold-start tool.
CHANGE THE LABEL BUDGETType plain English at a mouse CT scan, and a 3-billion-parameter model on a laptop drives the viewer through tool calls while the tools produce every number. I benchmarked five ways of getting structured output from small local models. Grammar-constrained decoding made every call valid and still came out 11 points behind the models' built-in tool calling, because they padded their answers with actions nobody asked for. A schema repair took the 8B model from 89% to 96%.
REPLAY THE SESSIONA doctor's visit conversation becomes proposed changes to the patient's medication list, each linked to the sentence that justifies it, and nothing is filed until the clinician signs. Before the visit, the patient reads their pill bottles to a voice assistant that only asks about what the chart can't settle. The language model only pulls out what was said; a rules engine decides. On 140 held-out synthetic visits the pipeline found 99.4% of the medication changes; one big prompt found 43%. Speech recognition and voices run on CPU.
TRY THE LIVE DEMOMost of what I build is really a way of testing something I already assumed was true. The ambulance project started as "relocating units helps, let's measure how much" and ended with the opposite answer: just parking the same eight units in better spots beat moving them around all year. The self-play project started as "the generator will learn to sit right at the edge of what the solver can do" and ended with me taking back three of my own claims, each time because of a test I wrote to try to break the previous one.
That's why I care more about the checks than the results. Knowing the exact answer to compare against, having a control matched on difficulty, writing tests that fail when a claim stops being true: that's the part still worth something a year later.