What 'distilling' an AI model actually means and why it matters to self-hosting open LLMs

Published Aug 10, 2026, 7:00 PM EDT Anurag is an experienced journalist and author who’s been covering tech for the past 5 years, with a focus on Windows, Android, and Apple. He’s written for sites like Android Police, Neowin, Dexerto, and MakeTechEasier. Anurag’s always pumped about tech and loves getting his hands on the latest gadgets. When he's not procrastinating, you’ll probably find him catching the newest movies in theaters or scrolling through Twitter from his bed. I find it fascinating that we can spin up an entire AI model on something as meager as a laptop. I can have a coding model running on the same device I use for Google Chrome. Frontier models generally require massive data centers packed with accelerators, so the fact that a smaller version of the same technology can run locally and not completely suck is genuinely impressive. One technique that helps make these capable local models possible is distillation. It’s a process in which a larger model teaches a smaller model to mimic its responses. The student remains much smaller than its teacher, but learning from those outputs makes it more capable and accurate than it would be with conventional training alone. Distillation teaches a smaller model how a larger one responds It’s trained to reproduce the patterns found in the teacher’s output Distillation transfers some of the capabilities of a large “teacher” model to a smaller “student” model. The student doesn’t receive a compressed copy of the teacher’s weights. It’s trained to reproduce the patterns found in the teacher’s outputs. Consider an image recognition model being trained to identify a fox. A conventional dataset provides the student with a simple label indicating that the image contains a fox. A teacher model can provide a probability distribution across all the available options, such as an 80% chance of a fox, a 15% chance of a dog, a 4% chance of a cat, and a 1% chance of a raccoon. Those probabilities contain more info than the final label because they show the student how the teacher distinguishes related objects. This is classical knowledge distillation, and it requires access to the teacher’s internal output probabilities, commonly called logits. A company can use this method when it owns both models. Covert distillation of a closed model works differently. A developer can send thousands or millions of carefully constructed prompts to the teacher, save the responses, and use them as training data for another model. Older models could sometimes be prompted to produce detailed reasoning traces, while frontier labs increasingly hide or obfuscate those internal traces. A distiller can still collect final answers, tool-use examples, code, generated explanations, preference judgments, and grades assigned to other responses. This is usually described as output-based or black-box distillation, although the boundary between distillation and ordinary synthetic-data fine-tuning is blurry. My colleague at XDA has done a much smaller version of this with a local 7B model. They generated 30,000 synthetic conversations based on Home Assistant documentation, with each example pairing a user request with the correct YAML automation. One example might ask the model to turn on the living room light at sunset, followed by a complete and structurally correct Home Assistant automation. They then fine-tuned the model using those conversations. Many local LLMs are distilled from larger teachers Whether that's Google Gemma or Deepseek Some of the most important models in the self-hosting world were built by learning from much larger models. Stanford’s Alpaca was an early example of output-based imitation. Researchers started with Meta’s LLaMA 7B and fine-tuned it using 52,000 instruction-and-answer examples generated by OpenAI’s text-davinci-003. Google has used a more conventional form of knowledge distillation within its Gemma family. The company trained the 2B and 9B versions of Gemma 2 using predictions from a larger teacher model. Google reported that the resulting models were competitive with models two to three times their size, although it didn’t identify the teacher model in its technical report. DeepSeek also used distillation extensively across its R1-Distill family. The full DeepSeek-R1 has 671 billion total parameters, with 37 billion active per token. It's not something normal consumer hardware can handle. DeepSeek used reasoning data generated by R1 to fine-tune smaller models based on Qwen2.5 and Llama 3, releasing versions ranging from 1.5B to 70B parameters. DeepSeek later distilled the improved R1-0528 into Qwen3 8B. The resulting model can be downloaded through Ollama, LM Studio, llama.cpp, or Docker Model Runner and run on ordinary consumer hardware. DeepSeek says the distilled models developed stronger reasoning capabilities than similarly sized models trained solely with reinforcement learning. Distillation is how local LLMs punch above their weight It's also how you have a capable model running on your laptop Distillation is only one part of what makes local LLMs possible. Small architectures determine whether a model can fit into memory. Quantization reduces the size of its weights and runtimes such as llama.cpp, Ollama, and MLX make those weights usable across consumer hardware. Distillation improves how much capability developers can fit into that limited parameter budget. Companies such as Meta, Google, Alibaba, and DeepSeek have enough resources to continue training models without distillation. However, we probably wouldn’t have as many genuinely useful small models without it. Devs will require much more data and experimentation to match the capability of a distilled model. The alternative would be accepting a smaller model that simply isn’t as capable. An 8GB laptop can technically run a quantized 3B or 4B model. The official 4-bit version of Qwen3 4B, for example, occupies around 2.5GB before it is loaded. That number doesn’t include the operating system, inference runtime, context cache, or any apps running in the background. Once all of that is accounted for, an 8GB machine has little headroom and can struggle even with a model that fits on paper. A 16GB device is far more practical for running quantized 7B or 8B models. Distillation allows those models to inherit behaviors that would otherwise be associated with much larger systems. Open LLMs are only going to get better Local AI has come a long way in just a few years, and there is plenty of room for further improvement. Models like Gemma 4 can already run on consumer hardware, while companies like Alibaba and Meta are building smaller models that deliver impressive performance without requiring massive amounts of compute.