How I A/B test LLM prompts without fooling myself

The author recounts an experience where they tried to determine if a new version of a support assistant prompt was better than the old one by running a small A/B test. Initially, the new prompt seemed to perform better, but the author soon faced complaints and had to roll back the change because the test wasn't statistically significant. This story highlights the importance of rigorous testing and sample size in AI development to avoid misleading results and ensure better overall performance.