Switching our LLM-as-judge from 5-class to binary in CI: the patterns we kept

The team recently revamped their LLM-as-judge system from a 5-point helpfulness scale to a binary one, significantly improving inter-rater reliability from a Cohen's kappa of 0.47 to 0.78. The shift to a simpler binary scale was key, as it streamlined the labeling process and allowed the CI pipeline to adapt more easily. This change highlights the importance of refining evaluation metrics to enhance consistency and reliability, ultimately benefiting the overall performance and accuracy of the system.