I ditched cloud AI for a local LLM in Proxmox, and my home lab finally makes sense

Published Sep 1, 2026, 6:00 AM EDT Maker, meme-r, and unabashed geek, Joe has been writing about technology since starting his career in 2018 at KnowTechie. He's covered everything from Apple to apps and crowdfunding and loves getting to the bottom of complicated topics. In that time, he's also written for SlashGear and numerous corporate clients before finding his home at XDA in the spring of 2023. He was the kid who took apart every toy to see how it worked, even if it didn't exactly go back together afterward. That's given him a solid background for explaining how complex systems work together, and he promises he's gotten better at the putting things back together stage since then. Local AI means different things depending on who's saying it, and it stopped being a crucial part of the conversation once we got semi-competent models we could self-host. I'd spent a lot of time thinking about home automation in terms of cloud vs. local, and I'd missed one big point. Where the model runs is a minor detail; what matters is who can see your automations and who gets to edit them. So I stood up Lemonade Server inside an LXC on my Proxmox node, pointed it at the custom four-tier model router I built for my ROG Flow Z13, and started asking every piece of automation in the house the same two questions. Can I read the rule? Can I edit the rule? Both need to be a yes before your home lab is really yours. Where the automation runs is the whole point The Proxmox API drew a line I didn't expect My Proxmox box is powered by a Ryzen AI Max+ 395 with 128GB of unified memory, and I've been mostly taking advantage of that memory to stuff as many LXCs and VMs onto it as I can. But that also means it's perfect for running a local LLM, and it's silicon I'm familiar with, as I run a Ryzen AI Halo and Z13 Flow, both powered by the same configuration. The other thing is that it was ever so easy to set up. It's a Debian 13 template, unprivileged, 250GB on the NVMe ZFS pool, 12 cores, and three device nodes mapped in: /dev/kfd, /dev/dri/renderD128, and /dev/dri/card0. The container sees the Radeon 8060S through the host's own driver. Lemonade Server 11.8.1 reports it as a gfx1151 device, and unified memory does the rest. It's a good thing I tested because Lemonade defaults to the Vulkan backend, and the ROCm backend is often better for performance. That didn't take long to fix, though. In the same container, ROCm runs a 120B model at 19 tok/s compared to 5 tok/s for Vulkan, and a 30B coder at 26 tok/s to 7.6 tok/s on Vulkan; so the swap was well worth it. It's not that long ago that ROCm was a complicated beast to set up, but now it's much more usable. The Proxmox API can't do everything I loaded up Claude to create the Proxmox LXC, because what's the point of setting up MCP servers if you're not going to use them? Except, the API decided that I needed to do some of it manually, because setting up device passthrough is clearly a level of access too high. 403 Forbidden: Permission check failed (configuring device passthrough is only allowed for root@pam) Even with root access, the API can't map devices into a container or set feature flags other than nesting. It only needed a couple of PCT commands from the shell, and it's probably correct not to let the API change hardware configuration for you. There was one other obstacle in my way, the RAM carve-out. When it's set to Auto in BIOS, this means 64GB: the host sees 62GB, the GPU gets 65,536 MiB of dedicated VRAM plus 31,967 MiB of borrowable GTT. That's a Windows-shaped default, but you don't need that when using Linux. The fix is to go in the other direction. Set the iGPU frame buffer to the 512 MB minimum, and use ttm.pages_limit in the kernel line to increase the VRAM carve-out to 100 GB. I hit one OOM-killed server at 99.99 GB of usage, and that told me I needed to unload models instead of keeping them all in memory. Bosgame M5 AI Mini Desktop $3000 $3600 Save $600 RAM 128GB LPDDR5X DDR5-8000 Strix Halo is a perfect choice for Proxmox to run on, with plenty of RAM to share between containers. CPU Speed Base 3GHz, Boost up to 5.1GHz What automation is allowed to touch is where the vendors go quiet Routing accuracy and answer quality are two different numbers Let's take an example from my house. My eero network intercepts all DNS traffic on port 53, even if I've got a local DNS set up. Each node is also a Zigbee hub, and Amazon auto-connects devices using credentials I don't have to type because the chain of automations links together for convenience. My container's version of that is the same set of MCP servers I already had: Proxmox, both Technitium DNS nodes, and Home Assistant. I can measure routing accuracy because I have a set of rules to put things on the correct models and another set of questions to trigger them. The eleven questions range from a one-line greeting to two hard-reasoning prompts, and my router put them all where they're expected. Answers are harder to quantify and even harder to make consistent. The 2B's first reply to "what's 2+2" was empty because it spent its whole token budget thinking, and the coder's fix for a divide-by-zero was to wrap the divide-by-zero in a try/except. I'd rather report both numbers than round them into one. The model router is what makes the lab make sense Except the judge LLM had been quietly lying to me The thing serving requests isn't a model; it's a policy. Lemonade's router recipe lets you register a collection of models and a rule set, and ours, unimaginatively named Squeeze, is four tiers: Qwen3.5 2B for quick hits, Qwen3.5 9B as the default and the only tier that gets images, Qwen3-Coder 30B-A3B for anything that smells like code or tool use, and gpt-oss-120b for hard reasoning or big pastes. A small model sits in front as the difficulty judge, and its entire job description fits in one prompt: reply EASY, NORMAL, or HARD. The rules are the point. Under 200 characters and judged EASY goes to the 2B. Anything over 12,000 characters, or judged HARD, wakes the 120B. A stack trace, a code fence, or a tool definition routes to the Coder. And if a client sets a metadata flag, it can pin the heavy tier regardless. Every response includes an x-lemonade-route header naming the rule that fired, and adding route_trace to the request returns the per-rule decision, condition by condition. Try asking your mesh router why it did anything. Except there was a problem The original 0.8B judge is a thinking model, and when asked for one word, it writes a plan instead and runs out of tokens before it says the word. The router's on_error setting quietly turned that into "not HARD," so every hard prompt fell through to the 9B, which answered well enough that nobody looked at the header. Swapping the judge for a non-thinking 4B and rewording the prompt fixed it, and the first hard prompt through the repaired router came back with the 120B's answer and the judge's own rationale attached: "The proof requires a mathematical argument about irrationality... involving deep logical reasoning." I didn't fix it because it was slow. I fixed it because I could read it. To be clear, this costs something. The 120B is slower than any cloud model, and the judge adds three or four seconds of fixed overhead to every request, which is why the 2B's router-level number is 46 tokens per second when the model itself is far quicker. That's the price of being able to read the rules. The commercial products charge it, too. You just never get the receipt. Control is one feature you can only self-host The industry will happily sell you local AI that runs on your network now, but it runs in a box, and you can't tweak it. It won't sell you the prompts, the rules, or the reasoning. And that's why I self-host my models for my home lab, because I want to write the harness around them.