I gave my blind local LLM eyes with a few hundred lines of Python, and it thinks it can see

Many of the open-weight models worth running locally still can't actually see anything, including, up until recently, DeepSeek-V4-Flash-0731. I had the latest DeepSeek V4 Flash model running across two DGX Sparks, and it's one of the most capable models I've ever used on my own hardware. Unfortunately, it's also completely blind, which can be pretty frustrating. If I wanted to ask it about an image or screenshot, I couldn't, and there wasn't an easy way around it. Instead, I built my own, in a way that slots into my existing workflows like with Hermes Agent. It's not a one-to-one replacement, as it uses a sidecar model to describe the photo to the model, but it works, and the fix was just a few hundred lines of Python that you can run as well. Even better, DeepSeek has no idea it's happening, and it just sees image descriptions that help it formulate responses. The proxy captions the image before the model ever sees it Your question decides what the caption says When you send an image through an OpenAI-compatible API, it arrives as a content block sitting in the messages array alongside your text, with the picture either as a URL or a base64 blob. A model server with no vision encoder won't typically know what to do with it, and what happens next depends entirely on the stack you run. Most will just ignore it, but some might fail on the request entirely, like with LibreFang. If you have a model that can't read images, then your next steps aren't often great. You can switch to another model that happens to have vision built in, which usually means giving up whatever made you pick your current one in the first place, you can wait for a multimodal variant of your current model, or you can use an unofficial version of the model with a merged vision tower. If you're really stuck, you might even run a separate vision model and copy and paste descriptions by hand. What I ended up doing instead was creating a proxy that understands the OpenAI-compatible API and handles communication itself. I moved DeepSeek off of port 8000 and onto 8010, where nothing but the proxy talks to it, and the proxy took over port 8000. Every client continues working at the same address, and everything continues working. A normal POST request to /v1/chat/completions is opened up and inspected, alongside everything else like /v1/models and health checks being forwarded straight through. However, I added a separate step that also ensures the useful information is extracted. If you send a 4B vision model a screenshot of an error in a terminal, it might transcribe the error into text, but it might just say "A terminal with an error message written." It's perfectly factual, but it's also probably no help whatsoever to the model that now has to figure it out. To solve this, I made it so that the proxy sends your own message text into the caption request. The vision model is told that it's describing the image for another model that can't see images, that it should use the user's task to decide what matters, that it should transcribe visible text exactly, errors and labels and values included, and that it shouldn't attempt to solve anything. The model never had to know Hermes, along with everything else, kept running just fine, without needing to change anything on the client side. Hermes had been running with DeepSeek just fine, sending everything to :8000, and when the shim took over that port it kept working after that. It doesn't know the shim exists, and it doesn't need to. For example, I sent it a photo of a POS system, asking "What is this?" as the caption. Hermes then sent a regular request to the endpoint: {"role": "user", "content": [ {"type": "text", "text": "What is this?"}, {"type": "image_url", "image_url": {"url": "data:image/ png; base64,iVBORw0KGgoAAAANSUhEUg..."}} ]} Once it hits my DeepSeek deployment, the image block is completely gone, and a text block is sitting in its place: {"role": "user", "content": [ {"type": "text", "text": "What is this?"}, {"type": "text", "text": "[Image description]\nA black all-in-one touchscreen POS terminal on a wooden shop counter, branded \"eposnow\" on the base. A handheld barcode scanner rests in a dock in front of it...\n[/Image description]"} ]} The text block of what I typed passes through untouched and sits first, and the description generated by the other model takes the position the image used to occupy. DeepSeek, reading only that, came back with this: That's an EposNow touchscreen POS terminal, the black all-in-one credit-card terminal/till sitting on the counter, with a barcode scanner docked in front of it. The manufacturer branding you'd see behind that kind of unit is generally EposNow's own hardware line (it runs their Android-based POS software, with the payment reader built in so staff don't need a separate card machine). The vision model identified the logo from the front of the unit and identified the scanner sitting in its dock. it also identified other items in the photo, including the business I had taken the photo at (because a business card was visible on the table), meaning that it wasn't just a generic description that lacked detail. The best part is, though, that the model which wrote the response to me can't see. Image descriptions are generated at temperature zero with a fixed seed, and they're stored in a SQLite cache keyed on a hash of the image, the question, and the caption settings. An agent harness resends the entire message history on every single turn, so without a cache, a conversation with three screenshots in it will try to process all three images every time you send a message. Responses are returned without issue, returned as raw bytes without being parsed or re-encoded. That means reasoning traces, tool-call deltas, and stop reasons are seen by the client in the same form as whatever the LLM's server sent. You don't need two DGX Sparks for any of this The proxy never touches the GPU While I'm running this setup on two DGX Sparks, you don't need powerful hardware at all to run this. The Python script uses FastAPI, httpx, and uvicorn, and both of the upstream models it uses are just OpenAI-compatible URLs, meaning that you can use a combination of llama.cpp, Ollama, LM Studio, vLLM, or anything else that's OpenAI-compatible. The only important question is whether you can afford to keep a second model resident alongside your first. On my setup, the image model has a GPU memory utilisation of 0.07 while DeepSeek uses 0.78. There are plenty of smaller vision models you can run as well, like Moondream 2 or Qwen3-VL-2B, which use little RAM and can even run on a separate machine if you can't fit it all on one device. I'm using Q4 Qwen3-VL-4B, which uses roughly 6GB of VRAM and is noticeably better on documents and screenshots. There's an obvious caveat, though; if you're using a 4B text model, you probably don't need this at all, because plenty of good small models already have vision. This works best when your primary model doesn't have its own vision tower, or you don't have the available VRAM on the machine running the primary model to also load its vision tower. The only major gripe I have with this setup is that things get lost when it's a separate model interpreting an image; things like fine spatial relationships, text the image model didn't consider relevant, or anything it didn't know to look for. As well, if you send two images in the one message, they get described separately, one after the other, leaving any actual comparison between them to the model that can't actually see either of them. Shortly after I built this, DeepSeek announced (and subsequently released) V4-Flash-Vision-Exp, which attaches a vision encoder and aligner onto the same V4-Flash architecture. It's still a bit messy to run at the moment, but there's a steady stream of patches people are working on for all model servers, and given that it performs text-agent performance just as well while substantially improving multimodal work, I'd probably just go with that instead of this proxy. Still, many models don't have a vision tower, or you can load the models but without the vision tower enabled. This project lets you have an additional vision tower for any reason, running on another server if you need it to, and you can check it out on my GitHub.