We gated CI on six open-source LLM eval frameworks. Only two survived the merge queue.

The experiment involving six popular open-source evaluation frameworks for large language models (LLMs) revealed that only two could reliably pass through a continuous integration (CI) merge queue over eight months. The key factor for success was deterministic checks that produced consistent results, unlike the others which exhibited flakiness. This matters because it highlights the importance of reliable and consistent evaluation metrics in CI processes, especially as LLMs become more integrated into development pipelines. The findings suggest a need for more robust frameworks to ensure smooth integration and consistent performance in production environments.