Zero failures isn't zero risk: the rule of three for evals

The rule of three for evals says zero failures in N runs is a count, not a rate. With 0 failures in N independent runs, the exact 95% upper bound on the true failure rate is 1 - 0.05^(1/N), which 3/N approximates. After 100 clean runs you still cannot rule out a 2.95% rate, about 1 in 34. Here is the reading that bites you. Your eval harness runs the agent 100 times, prints "0 failures," and the tile goes green. Someone screenshots it into the launch thread. The unspoken translation is "the failure rate is zero." It is not what the data says. I wrote a small script to make the gap concrete, so I ran a real gate over 200 deterministic agent runs first, counted honestly, and got the dashboard everyone trusts: gate: spend= ln(0.05)/ln(1-target): target = the failure rate you want to be able to rule out at 95%. bound bound 1.49% Enter fullscreen mode Exit fullscreen mode Shuffle which scenario the gate sees first, run it twice, reorder the whole fixture: the count stays 0, so the bound is byte-for-byte the same. There is no effect size to permute, no ratio whose denominator is secretly doing the work. That is the opposite of the class of eval post that gets dropped, where the "finding" is really a fixture setting in disguise. Here the only thing I observe is a count, and the only thing I do to it is bound it. If you distrust the bound, you are distrusting the binomial, which is a fine thing to argue about and a very different argument from "he tuned the numbers." The whole thing runs in a second: Python 3.13.5, stdlib only, offline, no keys, no funds, three runs byte-identical, exit 0, empty stderr. Picking the right statistical test is a sibling problem, and I wrote up the paired-data version of it in why your A/B eval probably needs McNemar, not a two-proportion SE. Both belong to the same family of pre-execution gates for AI agents: decide what you can actually claim before you act on the claim. What does your board show The next time a tile goes green on "0 failures in N runs," do the one-line translation out loud. Zero in 30 is a rate that could be 1 in 11. Zero in 100 is a rate that could be 1 in 34. Neither is zero, and the dashboard will not tell you that unless you make it. I publish the runs that correct my own reading, not only the ones that flatter it, so follow along if that is your kind of thing. And a real question I don't have a clean answer to: when your eval last went green, did anyone write down the N, or did "0 failures" quietly get filed as "the failure rate is zero"? I want to know how many runs your team treats as enough, and why that number and not ten times it. AI disclosure. I drafted this with an AI writing assistant and edited every line; the framing, the gate, and the reading are mine. Every output block is pasted from one real local run on 2026-07-22. zero_failures.py sha256 d3b3f205909640fc…, run output sha256 cc7aca328fdd7236…. Recompute: python3 zero_failures.py prints the same bytes, exit 0, empty stderr, stdlib only.