A Better LLM Judge? The Rubric Made My Small Model Worse

In a quest to create a minimal LLM judge, the author experimented with a small model and a simple rubric, only to find it agreed with human judgments around 43% of the time. The model struggled to differentiate nuanced scores and overly simplified the grading. By separately scaling up the model or refining the rubric, the improvements were less than expected, highlighting the complexities in creating effective AI-based judges. This experiment underscores the challenges in developing tools that can truly replicate human judgment.