LLM-as-a-Judge: I Built One From Scratch, Then Checked It Against Humans

The article explores the creation of a model designed to evaluate other large language models (LLMs) by grading their outputs on a scale from 1 to 10. The author initially tests the model by comparing it to human judgments on specific labels, but soon finds that most LLM outputs lack clear labels for comparison. The experiment reveals that while the self-evaluation approach works surprisingly well at first, it eventually uncovers discrepancies that highlight the complexities and limitations of using LLMs to assess each other, pointing to broader issues in the reliability and oversight of AI systems.