Small Language Models: Rethinking What Intelligence Actually Requires

"Scale solves everything — until it doesn't." Introduction: A Result Nobody Predicted In March 2024, Microsoft published a technical report with a claim that most researchers found difficult to take seriously at first. Their new model, Phi-3 Mini, had 3.8 billion parameters. GPT-3 had 175 billion. GPT-4 is estimated at somewhere above a trillion. And yet Phi-3 Mini outperformed GPT-3 on standard benchmarks, approached GPT-3.5 on several tasks, and ran entirely on a laptop with no...