Gemma 4's quantized models finally made local AI practical in my homelab

Published Jul 28, 2026, 7:30 AM EDT Adam has a degree in Engineering and has a life-long interest in technology. He has been writing for over a decade for various print and online publications, with a focus on consumer tech. He joined How-To Geek in 2024 while working at Pocket-lint. As well as being a long-term fan of Apple products, he also has a strong interest in smart home tech, running a Home Assistant server at home to automate all his smart home devices. He believes that the ideal smart home should work with minimal interaction from the user, with automations running as if by magic rather than requiring you to push buttons on a control panel. You can find more of his work on Muck Rack. I run Home Assistant in Proxmox on my mini PC, along with multiple other services. I use local LLMs running in Ollama, but they're very slow on my local hardware. I've always wanted to use a local LLM to make my local voice assistant smarter, but until now, it's always been far too slow or inaccurate to be of any use. My local voice assistant has limited understanding Simple commands work well, but it doesn't really understand intent Credit: Adam Davidson/How-To Geek Home Assistant has its own voice assistant called Assist. By default, it uses a predefined set of built-in sentences and intents to turn spoken commands into Home Assistant actions. For example, you can say, "Turn on the study light," and Assist recognizes the pattern is for the turn_on action, and that the target is the study light. The light turns on in under a second. The problem is that if you step outside the limited set of predefined commands, Assist doesn't work. If you follow up a command saying, "Turn on the study light," with another command saying, "Turn it off again," Assist will throw an error. There's no entity called "it," so it doesn't realize that you're talking about the same light. You can solve this by hooking up Assist to an LLM. If a command is recognized, Assist will handle it directly, but if it doesn't understand what you're asking, the command gets passed to the LLM to try to figure out the intent, and the appropriate command is sent back. If you hook Assist up to a cloud-based service, such as OpenAI, it works really well and is more than fast enough to be usable. Using a cloud-based LLM isn't ideal, since your commands and data end up on third-party servers. You can hook Assist up to a local LLM, but until now, I've always found using a local LLM to be far too slow on my hardware. Qwen 3.5 4B is capable but slow Waiting five seconds for anything to happen is just too long I decided to try out a few of the local models I currently use to see how well they could do, before giving one of the quantized Gemma models a try. I started off with Qwen 3.5 4B, which is reasonably capable but still small enough to run on my hardware. The first simple test was whether it could turn on a light and then correctly respond to the command, "Turn it off again." Qwen 3.5 managed to perform the correct action, but it took five seconds before anything happened, which is far too long to be practical for a voice assistant. I then tried some commands that were more difficult to understand. These were the commands I wanted Assist to successfully complete: Turn on the study light Turn on the bedside lamp Set the first light you turned on to 40% Set the other one to 70% Turn off the dimmest of the lights you just turned on Set the other one to 50% Turn the last light you turned off back on Turn off the light that was most recently at 70% Qwen 3.5 got as far as the fourth command, but when I asked it to turn off the dimmest of the lights it had just turned on, it turned off the bedside lamp, which was the brighter of the two. Not only was it too slow, but it wasn't up to the task. Llama 3.2 3B was even worse It took 10 seconds for anything to happen Since the 4B model was too slow, I wanted to try a slightly smaller model to see if that would improve things. I tried Llama 3.2 3B next. This proved to be even worse. I started with the simpler test, asking to turn on the study light, and then asking, "Turn it off again." It took a full 10 seconds before the light turned off, which was almost twice as long as Qwen 3.5, and completely useless for a voice assistant. When I gave it the more complex commands, it failed at the first hurdle, being unable to correctly handle the command, "Set the first light you turned on to 40%." It had to ask for clarification as to which light I meant. Nemotron Mini 4B was faster but too dumb It couldn't handle proper reasoning The next model I tried was Nemotron Mini 4B. This is a small language model from NVIDIA that's optimized for function calling and lightweight enough to run locally. The initial results were very impressive. The simple test of "Turn it back on again" worked in about two seconds, which was much faster than either Qwen 3.5 or Llama 3.2. Things didn't go so well with the more complex test, however. It also made it only as far as the first genuinely complex instruction. When asked to follow the command, "Set the first light you turned on to 40%," it claimed that the first light it turned on wasn't controllable, which is a little odd considering it turned on just a few seconds earlier. This model wasn't going to cut it, either. Granite 4 3B was fast and did better One-second response time was good, but it ultimately failed Next on my list was Granite 4 3B, a small model from IBM. This model was very promising at first and breezed through the simple test, turning the light off in about a second, which is plenty fast enough for a voice assistant. Sadly, this model also failed when it hit more complex commands. When asked to "Set the brightness of the first light you turned on to 40%," it turned the bedside lamp to 40%, which was the second light it had turned on. It did perform the correct action on one of the lights that it had just turned on; it just failed to pick the right one. Gemma 4 E2B was the model that finally cracked it Fast responses and it could understand more complex intents Having failed with every attempt so far, I was nervous about the chances of the quantized Gemma 4 E2B model. This is the smallest model in Google's Gemma 4 family, with the E2B standing for Effective 2 Billion parameters. The full model is larger than 2 billion parameters, but its design allows it to run more efficiently than that total size might suggest. Gemma 4 E2B sailed through the simple test, turning the light back on in around a second. It then aced the more complex test, too. It correctly understood the intent at each stage and performed the appropriate actions. The response times were more than acceptable for use with an everyday voice assistant. Gemma 4 has made a local voice assistant possible Using Gemma 4 E2B with Assist still can't match the speed or capabilities that you get when using cloud-based LLMs. On my hardware, however, it finally allows me to have a local voice assistant that doesn't require me to remember the exact way to phrase commands in order to do what I want. I can be far vaguer, and Assist will (usually) still be able to perform the correct action.