My local LLM setup got faster after I stopped obsessing over parameter count

Published Jul 23, 2026, 1:00 PM EDT Nolen began their writing career in 2019, with three years dedicated to editing the Creative section at MakeUseOf. Their expertise lies at the crossroads of technology and creativity, covering areas like photography, video editing, and graphic design. Outside of work, you'll often find Nolen diving into a good book, writing their own stories, or playing video games. Local LLMs have earned their spot in a lot of setups, mine included. And you don't really need me to sell you on why offline inference is worth having anymore. But there's a decision that comes up that can tend to trip people up, which is picking the right model size for your hardware. If you're on workstation-tier gear or something with 24GB+ of VRAM to spare, sure, chase the big parameter counts. Consumer hardware is a different story though. Pushing a 20-something-B model on 8GB VRAM doesn't get you 20B-level results, it gets you a 20B model held together with system RAM and offloading, and that's not the same thing. I've been running local LLMs under 4B for a hot minute now instead of chasing bigger models because size doesn't matter as much as I used to think. Want to stay in the loop with the latest in AI? The XDA AI Insider newsletter drops weekly with deep dives, tool recommendations, and hands-on coverage you won't find anywhere else on the site. Subscribe by modifying your newsletter preferences! The B count trap Bigger models aren't the flex people think they are The parameter count is the first thing you see on a model card, and it's also one of the least useful things about it. The number tells you how big the model is on paper. It doesn't tell you if it'll actually run on your setup, or how it'll behave even if you manage to get it running on limited hardware. And there's a big difference between a model loading and a model actually being usable. Take Gemma 3 27B at Q4_K_M. It needs around 15GB of VRAM just for the weights, and that's before you factor in the KV cache, which grows with every token in your context. Even on a 24GB card you'd have to keep your working context modest to avoid running out of space, and on 8GB you're not running it at all without offloading half of it to system RAM. That drops you from 40 tokens per second down to something closer to 8. So yes, the model technically works, but it's basically useless for anything interactive. And the KV cache is the part nobody talks about. Llama-3-8B at 32K context adds roughly 4GB of KV cache on top of the weights themselves. Scale that up to a bigger model and the cache can genuinely rival the weight file in size. So a 20B with a 128K advertised context is just a marketing number instead of a guarantee. It just means the number was true on an 80GB A100 somewhere. What actually matters when picking a model And most of them you won't find in the model name Once you stop staring at the parameter count, the more useful question is what actually predicts whether a model will do the job. And there's more than one answer, but these are the ones I've noticed in my own local LLM workflows… Tool calling is probably the biggest gap nobody's checking. There's a benchmark called BFCLv3 that measures how reliably a model handles function calls, scored out of 100. Granite 4.0 H 1B lands at 54.8. Gemma 3 1B lands at 16.3. Same parameter count, and the Granite model is more than three times better at the thing that decides whether MCPs, function calls, or local agent workflows will actually work for you. Then there's the architecture itself, which I think gets overlooked too. For example, Granite 4 and Liquid AI's LFM2 both use a hybrid Mamba-transformer setup, and IBM claims it cuts memory use by up to 70% compared to a pure transformer at the same size. So two 1B models with the same B count can behave completely differently, essentially. Instruction following is the related but separate one. It's basically a measure of how well a model actually sticks to what you asked, and there's a benchmark called IFEval that scores it out of 100. Granite 4.0 H 1B lands at 78.5 here, which is essentially neck and neck with Qwen 2.5 32B at around 81. So a 1B model is holding your formatting rules about as reliably as one 30 times its size. One model per job beats one model for everything The specialist approach Once you accept parameter count isn't the point, then it's worth looking at your setup. Now that you don't have to find the one that handles everything you throw at it because of its high B count, you start building a small stable where each model actually earns its spot. The first job in the stable is routing. A tiny model that parses what a request is asking and picks which function to call, and the good ones here are well under 1B parameters. Liquid AI's LFM2.5-230M is a good example of this, which is well under a billion parameters. Then there's the execution of whatever the router hands off, which is where a model with strong function calling is useful. Qwen models are among the strongest open-weight LLMs when it comes to function calling, and Granite 4.0 H 1B fits here nicely too with its hybrid Mamba architecture. Vision is its own job because multimodal models are their own category, and there's no reason to keep one loaded until you actually need it. Gemma 4 E4B is the one I already had for this. Reasoning is the last one, and this is where you go bigger, but only as big as your hardware can actually run properly. The ceiling here will, again, be determined by your VRAM, not your ambition. The point is you compose these jobs into a workflow with separate models. Route with something tiny, execute with something purpose-built, and hand off to the generalist only when reasoning is genuinely needed. The under-4B lane is all you need I think the "bigger is better" instinct with local LLMs is one worth unlearning, especially if you're on consumer hardware. A model you can actually run properly, at a decent speed, with headroom for your context, is going to serve you better than one you're forcing to fit.