These small local models keep saving my 8GB GPU from an upgrade

These small local models keep saving my 8GB GPU from an upgrade

Published Nov 17, 2026, 6:00 AM EST Nolen began their writing career in 2019, with three years dedicated to editing the Creative section at MakeUseOf. Their expertise lies at the crossroads of technology and creativity, covering areas like photography, video editing, and graphic design. Outside of work, you'll often find Nolen diving into a good book, writing their own stories, or playing video games. Every few weeks a new open-weight model drops, and the version that all the reviews are about isn't usually one I can run on 8GB. The comments under those releases barely change either; someone always says to save up for a 16GB or 24GB card. But honestly, I'd rather invest that in something like a tablet because the rest of what I do on my PC truly doesn't need more than 8GB, at least not right now. So most of my heavy work still goes to cloud models, like big research projects, design work, building things, and so on. Claude isn't leaving my stack any time soon, probably not Gemini either. But I still use local pretty often, it comes in handy for anything I'd like to keep off data centers, like contracts, legal papers, financial docs, and health-related information. These are pretty small jobs that can be handled by small models. Gemma 4 E4B One of the most reliable AI chatbots This is only about my millionth time covering Gemma 4 E4B, because you genuinely can't go wrong with it. Google released Gemma 4 in April 2026, and E4B is one of the two small models in the family built for phones and laptops. The E stands for effective parameters, so it has 8B in total but a lot of those sit in per-layer embedding tables that only get used for quick lookups, so it runs more like a 4.5B model. I run it at Q4_K_M in LM Studio, and it's the only one here that can read images and handle audio input too. The catch with small models like this is what it knows. Google's own benchmarks put E4B at 69.4% on MMLU Pro, a test that leans on general knowledge, against 85.2% for the 31B. So its training data is decent but I'd still verify the facts. It also has native function calling and it's better than I initially gave it credit for. It calls my web and filesystem MCPs just fine (though Qwen models are still better at it). This is the model I use for personal research, or rather, synthesizing information I give it. It's especially useful for basically dumbing down complex health reports or contracts, at least the parts that tend to go over my head. Since it has vision, I don't even have to load in full documents or copy-paste large chunks of text, and can just quickly take a screenshot of the document and upload it to the chat. DeepSeek R1 Distill Llama 8B It's old but handy Yes I know, this one is considered ancient in local LLM timelines, but I still find it useful. DeepSeek released it in January 2025 alongside R1, its big reasoning model. It's Meta's Llama 3.1 8B, retrained by DeepSeek on around 800,000 examples of R1 working through problems step by step, so it picks up R1's habit of thinking out loud before it answers. A Qwen-based 7B version exists too, and it actually scores higher on DeepSeek's math and science benchmarks, though the Llama one does a bit better on coding. I run it at Q4_K_M which is about 4.9GB. This model gets my personal problems that still need to get worked out, like budget math or comparing anything that comes with numbers attached, basically, because its strong suit is math and logic. Despite this, I also find it's pretty strong at reading through a pile of messy notes and finding the one cause behind a few separate issues. The thinking block is where I get the most out of it, since I can check every step it took before trusting the answer. FluentlyQwen3-Coder-4B Not every tiny tool needs a cloud model Thanks to the concept of vibe-coding, I, too, have been embracing building my own little tools despite not writing a single line of code. And this model is perfect for it. FluentlyQwen3-Coder-4B is a community fine-tune from Project Fluently rather than an official Qwen release. It's built on Qwen3-4B and trained on NVIDIA's OpenCodeReasoning dataset, and at Q4_K_M it's 2.38GB, the smallest file in this setup. My use case for this model is pretty simple: I use it to help build small tracking tools for myself, like a simple HTML page for logging health test results. What I'd be logging is the private part, and I'd need to describe that to the model to get the basic form built, which is why I keep these little tools local. It actually also handles fiddly jobs around my notes in Obsidian. Cleaning up YAML frontmatter on journal entries is most of it, plus the odd regex for pulling dates out of tracking notes. As a general chat model it's pretty useless, and since it's a community fine-tune with no published benchmarks, I only hand it single-file jobs I can check by opening the result. A bigger card would be nice, but I don't really need it If someone handed me a 16GB card tomorrow for free, of course I'd accept it in a heartbeat. But I'm the type of person who uses what they have until it becomes unusable, and there's functionally nothing wrong with my PC. This doesn't have to mean subpar private AI workflows though. If you build the right stack of small models suited to your needs, and tune them right, that's really all you need.

Original Source

Read the full article at Xda-developers →

KhanList aggregates and links to publicly available news content. We do not host full articles from third-party sources. Always verify important information with original sources.