I ran local AI models on my PC without knowing what the acronyms mean, and they save me hundreds on cloud subscriptions

Published Sep 3, 2026, 12:31 PM EDT Abhinav pivoted from a career in banking to pursue his first love in writing. Even while working full-time, he continued contributing as an editor-at-large, a role he has held for more than 7 years. A lifelong tech enthusiast who has built three gaming and productivity powerhouse PCs since 2018, his passion for technology keeps him closely following the semiconductor industry, from NVIDIA and AMD to ARM. His MSc dissertation explored how artificial intelligence will reshape the future of work, reflecting his curiosity about the wider social impact of emerging technologies. Local AI inference is hugely popular at the moment, and open models have found a home in the workflow of most developers, analysts, creative workers, and home lab hobbyists alike. After all, who doesn't want an LLM that's limitless, free to use, can be run privately on your own hardware, and potentially save you hundreds of dollars on cloud subscription fees? Unfortunately, though, most of the discourse surrounding local AI models on community forums and subreddits tends to push more casual users away than to draw them in, while beginner guides simply assume a level of knowledge most users don't have regarding a technology that's still rather nascent. I too, found the terms intimidating before deciding to dive head-first into local AI inference, and most of what I know comes from my usage, experience, and tinkering alone. If you're in the same boat as I was, here's a way to get started. The first step is knowing your hardware The only acronym you need to know to get started is what "VRAM" stands for If you were to go through a local AI model hub like Hugging Face, you'd notice strings like Q4_K_M, IQ4_XS, GGUF or MTP quite often. More often than not, they become part of the reason why most beginners give up before picking a model. The great news about this is the fact that you can completely ignore most of that at step one. The first thing you need to familiarize yourself is your own hardware, wherever you're planning on running the models on. The primary component for all local AI inference is your GPU, and the part of it that matters the most is your VRAM. The VRAM is the memory that sits on your GPU, and it has to "hold" the model along with the working memory that's built as it runs. If the size of your model exceeds your VRAM, your GPU will offload the overflow to your system RAM, at which point, the speed of inference (measured by tokens per second) drops sharply. In that way, VRAM sets the ceiling on what you can run well. For example, my RTX 4070 Ti Super with 16GB of GDDR6X memory can comfortably run 13B-class models (models with 13 billion parameters). On the other hand, my Lenovo Legion 5 Pro on a mobile RTX 3070 with 8GB of VRAM is much better suited to the 4B-7B tier, which leaves room for the model to work. There are other variables and adjustments that can make running larger models run comfortably well on a smaller VRAM budget, but at this step, as a beginner, it's best to stick with what your card permits before you get the hang of it. The next step is choosing a "harness" Which is how you'll interact with your models Before you get anxious, a "harness" is simply the software you choose to run and interact with a local model on. It's the software you install, open, use to load a model, and then use it as you would. Two of the easiest options are Ollama and LM Studio. I'm personally fond of Ollama, because it provides just the right mix of command-line approach to interaction and also allows similar controls in its GUI, while taking away most of the burden that is associated with setting things up. LM Studio is easier still, with a more navigable GUI and a built-in model browser that lets you find, download, load, and chat with models. There is, however, something that even beginners should take note of. Ollama and LM Studio both make a lot of decisions on your behalf regarding inference, and while they are sensible and conservative for your hardware tier most of the time, they aren't universally optimal for every model or PC. Newer optimizations can also take time to arrive, meaning a convenient front-end has a convenience cost of leaving some performance on the table. But that's not a reason to avoid these tools together. On the contrary, I would recommend them to beginners who are just starting out, as I got started with Ollama myself. Once you start running into their limits, you can move on to llama.cpp or vLLM to get some more control over how your models run. Finally, you pick a model that complements your workflow Would you like a coder? A general assistant? Or something to handle your notes? When you know what models your hardware can handle, it's time to pick one. When it comes to local models, bigger isn't always better, and a model's capability often matters more. For example, a smaller model that's trained for coding can outperform a larger general-purpose model when you're writing Python scripts, and only be a worse choice for everyday conversations. As such, it's prudent to start by understanding what your workflow really needs. For coding, I am particularly fond of the Qwen family of models developed by Alibaba Cloud, and I can comfortably run Qwen3.5-9B at 87.76 tokens per second, which is fast enough to write Python scripts within seconds on my 4070 Ti Super, but this, of course, depends on the complexity of the prompt. For general purpose assistance, I like to look at Gemma 4 variants, which come in various sizes and can be run on a wide variety of hardware. Many Gemma 4 models also come with native vision and agentic capabilities, but you'll have to check if your specific model supports them via the model hub. Besides that, you'd also want to have a look and check the model's page to see if they support "reasoning" (sometimes referred to as "thinking"), and that means the model works through a problem before answering, which helps multi-step tasks at the cost of speed and tokens. Some models also support tool calling, which means you can rely on them for web search or code execution if your harness supports them. At this point though, it would be massively helpful to know a few basic facts about model quantization. Quantization is how the same model gets shrunk to fit different VRAM sizes. Whereas a model's full-precision weights can be quite large, quantized versions compress them at a small quality cost, and the tag on the downloads (Q4_K_M, Q8_0, IQ4_XS) tell you the kind of compression that was used. Q4_K_M means 4-bit quantization, and it roughly halves the size when compared with Q8 with minimal impact on quality for many models. Context length also matters when you're dealing with smaller VRAM sizes, and it also affects how much VRAM the model needs beyond its weights alone. The harness defaults, such as Ollama's, are usually a good starting point for most beginners. Hosting your own local models can save you money If you rely extensively on AI tools for your workflow, and you've got the hardware to host your local models, there's practically nothing to lose from running a model. For my coding workflow, I would frequently go over my Claude Code usage limits, but introducing Qwen 3-Coder 30B at Q4_K_M quantization and 16K context length in my workflow (averaging 56 tokens per second) allowed me to reduce my dependence on the cloud AI provider and saved me a lot of $5 top-ups every time I went over the limit so that I'd not have to wait for five hours to continue working.This is exactly the kind of advantage you unlock with running LLMs locally, and very little of the process requires you to obsess over the technical bits until you're ready to delve into it yourself.