Power analysis for LLM evals: how big does your eval set need to be to catch a 5% regression?

TL;DR: Most eval sets are sized by "what we had lying around", not by what they can actually detect. If your eval set is 50 traces and you are trying to catch a 5-point drop in pass rate, you are underpowered: the regression hides inside sampling noise more often than not, and you ship it green. A two-line power calculation tells you the size you actually need, and ours said roughly 4x what we were running. The number nobody computes We argue about which metric to use and skip the pr...