I checked six LLM-as-judge tools against human labels. The scoreboard was the wrong thing to read.

The study reveals that most tools for judging large language models (LLMs) focus on speed rather than accuracy, which is misleading. Instead of ranking tools by how quickly they provide a score, the author evaluated them based on how well they align with human-labeled data to ensure trustworthiness. The analysis highlights inherent biases in LLM judges, like favoring initial inputs or verbosity, underscoring the need for more rigorous validation against human standards. This matters because it emphasizes the importance of reliability over speed in evaluating LLM performance.