I ditched Claude's built-in search for a local embedding model, and my context window finally stayed clean

Credit: Nick Lews / How-To Geek Published Aug 31, 2026, 12:30 PM EDT Nick Lewis is an editor at How-To Geek. He has been using computers for 20 years --- tinkering with everything from the UI to the Windows registry to device firmware. Before How-To Geek, he used Python and C++ as a freelance programmer. In college, Nick made extensive use of Fortran while pursuing a physics degree. Nick's love of tinkering with computers extends beyond work. He has been running video game servers from home for more than 10 years using Windows, Ubuntu, or Raspberry Pi OS. He also uses Proxmox to self-host a variety of services, including a Jellyfin Media Server, an Airsonic music server, a handful of game servers, NextCloud, and two Windows virtual machines. He enjoys DIY projects, especially if they involve technology. He regularly repairs and repurposes old computers and hardware for whatever new project is at hand. He has designed crossovers for homemade speakers all the way from the basic design to the PCB. Nick enjoys the outdoors. When he isn't working on a computer or DIY project, he is most likely to be found camping, backpacking, or canoeing. Sign in to your How-To Geek account If you've ever had a long session with Claude Code or Cowork, you've probably had your context compressed before. Once that happens, Claude starts to lose track of the things you've already worked on. When you reference something in your code, like the "retry logic" section, Claude is going to have a harder time specifically finding that segment of your code. If you pay attention to what it is doing, you'll often see it run multiple commands trying to search for strings that turn up the correct portion of the code. It works, but it is clunky and a waste of tokens. And if you're using Fable, you really don't want to be wasting tokens. So, I paired Claude with a local embedding model that indexes my projects and provides Claude with a semantic search as an MCP tool. Rather than trying to find the correct keyword, Claude can just find whichever part of the code it's looking for using a semantic search. What does the local model actually do? Convert the code you're working with into a database for Claude An embedding model can be used to create a searchable map of your codebase. Practically speaking, that means that similar meanings are close to each other in the embedding, so casual phrases that are semantically similar to what a code explicitly says (or does) will link to the code, even if you don't use a phrase that explicitly matches something in the code. So, I decided to combine a local embedding model with Claude (via MCP) so that Claude can search for parts of the code using natural language. You can do that sort of thing with a frontier model, but it has drawbacks. Indexing a whole large codebase requires thousands of embedding calls, and you'll need to re-index frequently as the code changes. Doing that through a paid API creates an unnecessary recurring cost; doing it with a small local model is fast and only costs as much as the electricity does. There are a ton of local embedding models available today that fit almost any hardware—some of them only require a few hundred megabytes of memory, while others use several gigabytes. All-MiniLM is tiny and comes in at around 22 million parameters; EmbeddingGemma has 300 million parameters and is a bit smarter. I've been using nomic-embed-text since it is supremely lightweight and runs on a CPU, but I think I'm going to swap it out for a larger, smarter model eventually. It is fast and responsive, but Claude would probably benefit from a more accurate model, and I have the spare power available. Why do regular searches run into problems? They're very literal Claude's built-in search is pretty literal. It works well if you (or Claude) know exactly what to search for to find what you're looking for, but it effectively needs to guess if you only have a concept rather than a specific term or phrase. If you've ever asked Claude Code or Cowork to find something you did a few weeks ago during a session, you've probably noticed it repeatedly guessing at the right keyword to dig up some part of the codebase. That is where the ability to semantically query your codebase using a local AI can make a difference. Intelligent search benefits messy code more, but requires more upkeep Claude tends to be pretty good about naming parts of its own codebase so that it can easily search through the code. With legacy—or older human-written code—using Claude is a very different story. It is often quite messy. I have dozens of small programs I've written over the years that are poorly documented and basically held together with duct tape and luck. Those are the situations where a more intelligent search really shines. It is especially useful if the codebase is too large for Claude's context window—when you reach that point, Claude becomes noticeably less intelligent. There is also the maintenance requirement. As you modify code, the embedding becomes outdated. That means that you need to periodically rerun the embedding model to ensure that outdated results don't point Claude in the wrong direction. Local AI is a great complement to cloud-based AI models Local models aren't going to be nearly as "smart" as the frontier, cloud-based models, but that doesn't mean they're not useful. When latency, volume, and privacy are more important than pure intelligence, they can be very competitive. You just need to tailor the AI use to the job at hand and combine multiple smaller AI to create smarter systems. I've improved how effectively Claude searches code that I import and minimized the amount of clutter it generates in context. The next feature will be a feature that automatically runs the embedded AI periodically to ensure the index is kept up to date.