Quantum programs give random outputs, so a test has to check the pattern across thousands of runs. This does that for programs in Quantinuum's Guppy language. Pick a program below and plant a bug.
Each program is checked two ways: measured normally (Z), and after rotating the qubits (X). Some bugs only show up in one of them.
Picture a CI job for a quantum team. Someone edits a program, the job runs it on an emulator a few thousand times, and it has to say pass or fail. If it fails correct code at random, people learn to hit rerun. If it passes broken code, the bug costs real time on scarce hardware. Shotgate is that CI job.
Each test states what a correct program's outputs should look like, for example "00 half the time, 11 half the time, nothing else." The gate measures how far the real outputs are from that, and puts a guaranteed error bar around the number.
Every test has two thresholds. Below allow is drift a correct program can show from noise. Above detect is a bug. The gate fails only when the whole error bar is above allow, and passes only when the whole bar is below detect. Each mistake is held to 1 in 1,000 per check.
It runs 250 shots, looks, and doubles until the answer is clear, up to 16,000. An obvious bug fails after 250. A close call takes more. Looking after every batch is paid for in the error budget, so it can't inflate the false alarm rate.
The replay above is one run per check. These numbers come from repeating every configuration hundreds of times on fresh shots, and comparing against the two tests people usually write: run 1,000 shots and assert that no impossible outcome appears, or that every share is within 2 points of what it should be. Every value here is read from the recorded results when the page loads.
Selene is a Rust emulator with Python bindings, so it can't run in a browser tab and this page plays back recordings. The code that made them is on GitHub, where it installs with pip, runs its tests on every push, and exits non-zero when a quantum program regresses, the same way a failing unit test does.
guppylang and selene-sim from PyPI. Tested on Apple Silicon macOS and on Ubuntu x86-64 in GitHub Actions.@guppy function that reports its measurements, state what a correct run looks like, and add it to a suite.calibrate measures how far a known-good program drifts under a noise model and writes per-test budgets, like a snapshot file.Every shot came from Selene on one laptop. Real hardware has noise that isn't uniform, drifts over a day, and depends on how ions move around the chip. A budget calibrated here would need recalibrating against a real machine's noise.
The gate's guarantee is only as useful as the allow and detect values you give it. Here, detect is allow plus 6 points. A bug smaller than detect but bigger than allow can go either way, and the gate reports that zone honestly as inconclusive rather than guessing.
Splitting the error budget evenly across the looks, and the L1 bound used for whole distributions, both cost extra shots over a tighter design. I chose bounds I could check line by line over the tightest ones available.
Clopper-Pearson intervals, the Weissman L1 inequality and group-sequential testing are all textbook. What's mine is applying them to Guppy and Selene as a CI gate, the noise calibration, the planted bugs, and measuring where single-basis tests go blind.