ROC CURVE + OPERATING POINTS
X = FALSE-POSITIVE RATE
Y = TRUE-POSITIVE RATE
Y = TRUE-POSITIVE RATE
01THE TOOL · EVERY CLAIM TRACED BACK TO AUDIO
Read the report,
then listen to what it is citing.
Click any claim and it shows you the transcript turns it came from. Switch on an injected error and the clean verdict is replaced by what the detector actually said about the corrupted version.
0Caught this session
0Missed this session
ASSEMBLING TRANSCRIPT AND CLAIMS
ORAL ARGUMENT TRANSCRIPTAUDIO OFFLINE
00:00 / 00:00
Audio is not included for this case. Transcript evidence remains fully interactive.
GENERATED BENCH REPORT0 CLAIMS
Keyboard: space toggles playback, left and right seek five seconds, J and K move between claims. Audio remains at the selected evidence while the text shows the detector’s recorded judgment, not a live re-run.
02PLANTED-ERROR BENCHMARK · KNOWN GROUND TRUTH
Eight different ways to be wrong,
including one you can only hear.
Track A takes claims that are properly grounded and changes exactly one fact in each. I lean on speaker misattribution because the audio tells you who is talking, and a text-only benchmark has no way to check that.
RECALL + SEVERITY ACCURACY BY CLASS
ORANGE = RECALL
INK = SEVERITY
INK = SEVERITY
SUPPORT SCORE DISTRIBUTION
CLEAN VS PERTURBED
10 BINS, 0 TO 1
10 BINS, 0 TO 1
RETRIEVAL + METHOD ABLATIONS
| System | Precision | Recall | F1 |
|---|
03DECISION RULE AUDIT · SAME MODEL, DIFFERENT OPERATING POINT
Moving the threshold changed the numbers
more than changing the model did.
Analysis not available in this bundle.
ROC AUC · SUPPORT-SCORE RANKING—REAL, MODEST SEPARATION
SAME JUDGMENTS, THREE RULES
GROUND-TRUTH COUNTS INCLUDED
| Decision rule | Precision | Recall | F1 | FPR clean | Bal. acc. | Counts |
|---|
THRESHOLD SWEEP
| τ | Precision | Recall | F1 | FPR clean | Bal. acc. | Counts |
|---|
Held-out threshold
The threshold is selected on one half of the cases, then reported without retuning on the other half.
| Split | Precision | Recall | F1 | FPR clean | Bal. acc. | Counts |
|---|
Evidence sensitivity
If the detector is really reading the retrieved evidence, then shuffling those turns should change what it decides. So I shuffled them.
| Evidence | Precision | Recall | F1 | FPR clean | Anchor hit | N |
|---|
Recall at the selected threshold
Per-class recall at the held-out operating threshold, with sample size shown for every perturbation.
Severity confusion
Rows are planted severity. Columns are the detector’s assigned severity.
—
ACCURACY · N = —
| Truth Pred. | Minor | Major | Critical |
|---|
CONTROL-SET VERIFIER AUDIT
I tested the gate before relying on it.
—
Before regating
—
After regating
—
Drop rate
| Verifier prompt | Clean accept | Corrupt accept | Separation | Parse failures |
|---|
04ROUGE VS FAITHFULNESS · ORGANIC SUMMARIES
Word overlap only tells you a summary
sounds like the transcript.
···
Track B has not run yet.
Organic-summary results will appear here when the bundle contains model summaries, correlation points, and a leaderboard. The planted-error benchmark above is complete.
ROUGE-L VS FAITHFULNESS
EACH POINT = MODEL × CASE
—
CORRELATION
A summary can score well on ROUGE by reusing all the right names and phrases while pinning them on the wrong speaker, dropping a hedge, or flipping a claim around. Overlap only tells you the wording is familiar. What I wanted to know is whether each sentence still holds up against the transcript it supposedly came from.
| Rank | Model | Faith. | ROUGE-L | Delta |
|---|
05METHOD · TWO TRACKS, SEPARATE JUDGE
Start where I already know the answer,
then try the messy version.
TRACK / A
Controlled perturbations
Grounded claims are changed one dimension at a time across eight error classes. A verifier model different from the detector gates the clean controls, so the benchmark does not simply grade its own work.
TRACK / B
Organic summaries
Local models summarize held-out oral arguments without planted errors. Every resulting sentence is aligned to transcript turns and judged for support, contradiction, and speaker identity.
RETRIEVAL
Hybrid evidence search
Int8-quantised embeddings and BM25 each retrieve candidate turns. Reciprocal-rank fusion combines them before entailment and speaker-aware judgment produce the final evidence record.
Limitations
- A single local 8B detector cannot represent the range of current frontier systems.
- The corpus covers forty cases, not the full modern Supreme Court archive.
- Synthetic perturbations isolate errors cleanly but are not a complete model of natural hallucination.
- Severity labels are model-judged and should be treated as an operational rubric, not human consensus.