Why comparing average scores is the wrong way to evaluate LLM prompts (and what to do instead)

Most teams compare prompts like this: Prompt A average score: 6.8 Prompt B average score: 7.4 "B is better, ship it." I used to do this too. Then I ran the numbers properly and realized I'd been making deployment decisions on statistical noise. Here's what I learned about evaluating LLM prompts correctly, and the specific implementation I built. The problem with averages on small datasets LLM eval datasets are small. Most teams have 10-30 golden test cases. That's not enough dat...