My local LLM's real problem wasn't hallucination, it was a two-year blind spot

Published Sep 22, 2026, 7:30 AM EDT Maker, meme-r, and unabashed geek, Joe has been writing about technology since starting his career in 2018 at KnowTechie. He's covered everything from Apple to apps and crowdfunding and loves getting to the bottom of complicated topics. In that time, he's also written for SlashGear and numerous corporate clients before finding his home at XDA in the spring of 2023. He was the kid who took apart every toy to see how it worked, even if it didn't exactly go back together afterward. That's given him a solid background for explaining how complex systems work together, and he promises he's gotten better at the putting things back together stage since then. I went looking for hallucinations in my local LLM and found something worse. My setup is the thing I'm proudest of in my home lab: Lemonade serving models, Crush as the terminal-based harness, nothing leaving my network. But I've always wondered how much information those local models make up, so I put together a test suite to check. It barely invented anything. Out of 13 questions designed to bait it, it made something up once. The other twelve it was honest, and that was the real problem. My local LLM wasn't lying; it was two years behind Honesty turned out to be the problem My test script asks the same model the same questions in two ways: directly to Lemonade and through a self-hosted AI search engine. The same system prompt both times, telling it to say so if it didn't know. The 13 questions covered recent releases, facts that changed after training, exact version numbers, a control group of stable knowledge, and a list of fakes: a Firewalla router, a Proxmox subcommand, and a Crush flag that never existed. Without search, it refused 11 of the 13. Asked about the latest Lemonade Server release, the software running the model answering the question, it said it wasn't aware of the project. Asked what Crush from Charm is, the harness I was typing into; same answer. The fakes mostly worked too: it rejected the invented Firewalla router and the invented Crush flag every time. It did straight-up invent data once. When I asked about the qm autoscribe subcommand, which I made up, it explained that the helper reads a VM's configuration and writes a human-readable description in Markdown, HTML, or plaintext. Confident, formatted, fictional. That's worse than when I asked a fact about the iPhone 18 Pro launch, and it told me it didn't know because its knowledge cutoff was June 2024. It's not like I asked complicated questions either. Things like the current release number for Proxmox and Lemonade, which version of Caddy to use, and even what the current name of the Perplexica container was. These are the type of questions you'd ask when working, and if the model can't answer you might as well open a browser yourself. Self-hosting the fix wasn't difficult Perplexica is now Vane, and installing it was the easy part The self-hosted answer is Perplexica, renamed to Vane in March 2026, so older guides point at a Docker image name that no longer matches. It now bundles SearxNG, the privacy-focused metasearch engine that does the actual searching, so there's no need for a second container. I made a Debian LXC on my Proxmox node, installed Docker, and ran: docker run -d --name vane -p 3000:3000 -v vane-data:/home/vane/data --restart unless-stopped itzcrazykns1337/vane:latest Port 3000 opens a setup wizard, and Vane has a native Lemonade provider, so connecting my server meant pasting in its address. One trap: the URL needs /api/v1 on the end, or you hit Lemonade's web interface and the wizard saves a broken connection anyway. I wanted to use search inside Crush, where I work. Community Perplexica MCP servers exist, but they didn't work: Vane's current API wants a provider UUID plus a model key, where the old one took plain names. So I wrote about 60 lines of Python exposing one web_search tool. That hit another snag: the MCP Python SDK went 2.x and renamed the FastMCP class, so a fresh installation pulls a version your code can't import, and Crush reports only a closed connection. Pinning below 2.0 fixes it. The free search engines won't talk to you With everything connected, my first search returned nothing. Not a bad answer—no, because SearxNG's default engines were refusing it. DuckDuckGo served a CAPTCHA and Brave a rate-limit error with a 180-second suspension. A fresh self-hosted metasearch instance looks exactly like a scraper, because that is what it is. So I went through the engine list SearxNG exposes at /config and tested each by hand. Eight engines, six ways of saying no. Mojeek returned 403 to SearxNG's honest user agent but 200 to a browser one. Yep returned 403 to both. Brave rate-limited regardless. Google and Startpage failed silently. Exactly one engine worked: Bing. That's the part worth remembering. A self-hosted, private, independent search stack, and the only thing between it and nothing is Microsoft. Bing rate-limits too, suspending you for three minutes after a handful of queries, and Vane rewrites each question into about three searches, so you hit that wall fast. Three more unrelated faults sat beneath the blocked engines, and each presented identically: a query that never came back. Lemonade had autoloaded my model on Vulkan instead of ROCm, the same trap I hit on my Proxmox node, giving 10.7 tokens per second against 33, and 0.2GB of VRAM against 6.7GB. It was barely touching the GPU. Even at full speed, queries timed out because the model was a reasoning model with no output cap, and one answer ran past 15,000 tokens. Underneath it all, an environment variable that reverted every time I opened a new terminal. Vane Self-host an AI answering engine with Vane and stop paying for subscriptions. Search closes the blind spot, but only when everything lines up The model won't reach for the tool on its own I asked one question three ways: no search, through Vane as written, and through Vane with "use web search" on the end. The bottom row is what I built all this for: the right answer, sourced, from a model that had no idea two years of releases had happened. Mode Time Sources Answer No search 1.3s "the current stable Proxmox VE release version is 8.2" Search available 4.9s "I cannot provide... I don't have access to real-time information" Search requested 11.3s 10 "9.2, as indicated by the ISO file name proxmox-ve_9.2-1.iso dated May 21, 2026" The middle row bothers me. Handed a working search engine, it chose not to use it, then apologized for not having web access. That's worse than no search at all, which at least took a guess I could check. Left to itself the model searched on 7 of 13 questions; told to use web search, 12. The prompt doesn't switch searching on; it just makes the model more likely to use it, and you can't predict which questions benefit. Four models, four different failure modes Once I gave SearxNG a Brave API key, search was no longer the bottleneck. Every query returned 26 to 55 sources in about twelve seconds, and the results beat Bing's: the Proxmox wiki downloads page and the 8-to-9 upgrade guide, not the marketing homepage. So I ran the suite properly and hit a wall. Vane's research agent needs the model to emit structured tool calls, and none of my local models did it in a way the pipeline could use. That last entry is almost funny: it ran the search, got the sources, and said it cannot search the web. Model What went wrong gpt-oss-120b Too slow to finish, 42 seconds before the first token on a search prompt Qwen3.5 9B Spent its whole budget thinking, never produced an answer Qwen3 Coder 30B Printed raw tool-call markup as its final answer Qwen3 4B Searched, ignored the results, then said it couldn't search The one good answer came from the Coder model, on the Perplexica question itself. It searched, found the old image name still referenced everywhere, and said it couldn't identify the current one. Which is correct: the rename is recent enough that both names are live, and a confident answer would have been wrong. Honest beats useless, but current beats both I set out to prove my local model was making things up, and it mostly wasn't. It just didn't know the last two years had happened. Search closes that gap when it fires, and the 9.2 answer with ten sources proves the idea works. But it only fires when you ask; the free engines mostly won't talk to you, and four different local models couldn't drive the research agent. That's a thinner promise than "just self-host it" implies, even if the Web UI for Vane worked.