Benchmark Shadows Study: Data Alignment Limits LLM Generalization

A controlled study finds that data distribution, not just volume, dictates LLM capability. Benchmark-aligned training inflates scores but creates narrow, brittle models, while coverage-expanding data leads to more distributed parameter adaptation and better generalization. Benchmark Shadows: Why High-Scoring LLMs Can Be Worse at Real Tasks A new preprint, "Benchmark Shadows: Data Alignment, Parameter Footprints, and Generalization in Large Language Models," provides a controlled, em...