I gave Home Assistant a local LLM for voice control, then turned it off for most of what I say

Published Sep 20, 2026, 10:00 AM EDT Anurag is an experienced journalist and author who’s been covering tech for the past 5 years, with a focus on Windows, Android, and Apple. He’s written for sites like Android Police, Neowin, Dexerto, and MakeTechEasier. Anurag’s always pumped about tech and loves getting his hands on the latest gadgets. When he's not procrastinating, you’ll probably find him catching the newest movies in theaters or scrolling through Twitter from his bed. Home Assistant uses built-in intents, which are pattern-matched phrases such as “turn on the kitchen light” or “set the thermostat to 70.” These intents are parsed using sentence matching, with no model involved, which is why the whole process is so fast. I did not want to compromise that performance, but sentence matching starts to fail when you give Home Assistant a command that does not closely match one of its predefined phrases. To handle those failures, I added a local LLM to Home Assistant for voice control. I am using IBM’s three-billion-parameter Granite 4.1 Instruct model, but it remains turned off for most of my commands. Adding a local LLM to Home Assistant for voice control You need a small enough model here I run Granite 4.1 Instruct 3B runs through LM Studio. You don't need to do much to connect it to Home Assistant once you have downloaded the model — just enable the local API server in LM Studio, which will expose an OpenAI-compatible endpoint on a local network. Home Assistant connects to that endpoint through a conversation integration that supports custom OpenAI-compatible servers. I select Granite as the model and allow the agent to use Home Assistant’s Assist API. Granite 4.1 Instruct supports tool calling and structured JSON, which matter more here than its ability to produce elaborate replies. Home Assistant sends it the available tools and exposed entities, the model selects the appropriate action, and Home Assistant executes it. I then choose this conversation agent in my Assist pipeline and leave Home Assistant’s built-in handling as the priority. That way routine commands still go through the fast sentence matcher without reaching Granite. If the matcher cannot resolve a less predictable command, Home Assistant passes the transcript to the model instead. Why bother enabling it at all, then mostly disable it? Sometimes you are better off with intents Even a fast local LLM takes anywhere from several hundred milliseconds to a few seconds to respond. For something as simple as “turn off the lights,” waiting for a language model to think about the request is far worse than instant intent matching. Since 95% of my commands are simple and repetitive, I have little reason to send them through an LLM at all. I’m running the model on my NAS, which already has a lot of things running, doesn’t have the most powerful CPU, and has barely 16 GB of RAM. If I decide to run inference on every utterance, including “what time is it?” or “turn on the fan,” it would burn resources for almost no benefit. I also don't trust a local LLM with important commands because it's nowhere near as predictable as intents. Traditional intents get things right 100% of the time — “turn on the kitchen lights” will map to the same action without fail. LLMs, on the other hand, can misinterpret a command, hallucinate an entity name, or choose an unexpected action. For safety- or reliability-sensitive controls such as locks, garage doors, and thermostats, I always choose predictable intents. However, fuzzy requests such as “it’s kind of stuffy in here” or “I’m heading to bed” are where I let the local LLM take over. These kinds of commands require more interpretation, and the latter might imply an entire scene rather than a single action. This is the 5% of interactions where reasoning adds something that pattern matching cannot. The same routines can work with and without Granite You can always extend built-in intents Granite is most useful as a catch-all for commands Home Assistant doesn’t recognize. If one of those commands becomes something I say regularly, I can add it as a custom sentence instead of sending it to the model every time. Home Assistant lets custom sentences extend its built-in intents, while sentence triggers can run an automation or script when Assist matches a phrase. Also, a single pattern can include optional words, alternatives, wildcards, and number ranges, so I don’t need to list every minor variation separately. For example, the actual work behind a bedtime routine can remain inside a Home Assistant script. A custom sentence such as “good night” can call that script directly. Scripts exposed to an LLM also appear as callable tools, with descriptions that tell the model what each script does and when to use it. Granite can interpret a less predictable version of the same request and select the bedtime script, after which Home Assistant follows the exact sequence defined inside it. Local AI in Home Assistant Once you have voice control sorted, you can start looking at other ways to bring AI into Home Assistant. One of the easiest is to use it to build automations that would otherwise take a fair bit of time to put together yourself. You can also explore tools like AI Automation Suggester. It’s a custom Home Assistant integration that looks at your entities, devices, areas, and existing automations, then sends that information to an AI model.