What AI Engineers Get Wrong When Deploying Open-Source Models to Product

Since you’re reading this, I’m guessing you’re deploying open-source models to production and wondering if you’re doing everything correctly. What should you optimize? What could be overkill? Giving all you’ve got to a single LLM that does text generation and treating everything else around it as secondary infra may seem like a logical choice at this point. But in an agent, the opposite is often true. This surrounding infrastructure layer is where actually where most of the cost and latency accumulate. Lucky for you, this article will cover the four most common mistakes teams make when serving open-source models in production and how to fix each one. TL;DR: Serving open-source models in production rarely means running a single model. It often involves many small models running on shared hardware. Fix the retrieval layer, improve GPU utilization, and do the break-even math, and the choice between frontier and open models will matter much less than you think. The Four Mistakes at a Glance Mistake In production The fix Treating it as one big-model problem One model per GPU, low utilization, idle GPU memory, and you still get invoiced for it Pack many models onto shared GPUs with eviction. Overlooking retrieval A hosted embedding call in the request path in the 100ms range, and no reranking pass Self-host embeddings and add reranking. Confusing a runtime with a cluster Expecting vLLM to route, queue, and autoscale across many models Put a control plane above the runtime. Self-hosting at the wrong time A dedicated GPU sitting idle below break-even volume Rent the API until volume is steady. Mistake 1: You Treat It Like One Big LLM Problem Most teams carry the same architecture over from hosted APIs: there’s one model, and traffic flows through it. That works well for a single large generative model, but not so much for an agent. An agent actually relies on many small models: Embedder for search Reranker for precision Entity extractor and so on… Each request is short. The challenge here isn’t getting more out of a token. It’s switching between many models without leaving GPUs idle. If you’re stuck in your one-model mindset, you end up with one deployment per model, and each one holds onto a GPU whether or not it’s receiving requests. The fix is obvious: share the hardware. Load several models into one GPU, keep the frequently used ones in memory, and remove the least-recently-used (LRU) model only when another one needs the space. A single agent can call on the order of ten small models, and an organization can run many agents. So, one cluster can easily end up serving 20 to 40 models at once. Tip: Before buying a larger GPU, check utilization. A cluster running at ~20% just needs model packing and a shared queue, not more hardware. Mistake 2: Retrieval as an Afterthought If your embedding call runs in the hosted embeddings API, the added latency and cost aren't negligible after the demo. Retrieval-Augmented Generation (RAG) depends on this layer, and treating it as a single API call can make otherwise good agents slower and more expensive than they need to be. Embedding option p50 latency Cost to embed 1B tokens Retrieval quality (mean nDCG@10) Hosted providers (Voyage / OpenAI / Cohere) 84 to 180 ms $120 to $130 ~0.615 (baseline) Self-hosted (stella / Qwen3-Embedding-4B) 15 to 27 ms ~$10 to $17 ~0.600 (about 97.5% of the providers) Source: Superlinked, SIE vs hosted embedding APIs. The second part of this mistake is skipping reranking. Vector search returns candidates that are close but not necessarily the best matches, and a reranking pass over the top candidates improves precision. Self-host both steps (encode and score), and this layer is no longer a source of production mishaps. Mistake 3: You Confuse a Runtime with a Cluster This distinction here is easy to miss. vLLM, SGLang, and TensorRT LLM are runtimes. Their job is to take one model and produce as many tokens per second from a GPU (or a group of GPUs) as possible using techniques such as continuous batching and paged attention. These tools are good at that job by design. What they don’t provide, though, is the surrounding control plane: Routing requests to the right model Sharing a queue across workers Autoscaling from zero Managing the lifecycle of many models at once Now, that’s a cluster problem, and if you expect a runtime to solve it, you might as well write a scheduler yourself. Open-source options at the cluster layer include NVIDIA Dynamo, llm-d, KServe, and the Superlinked Inference Engine (SIE). vLLM (a runtime) A small-model cluster (e.g. SIE) Built for One large generative model across GPUs Many small models sharing GPUs Operations Generation Encode, score, extract, generate Many models on one GPU Not the design target Yes, with LRU eviction Scale to zero No Yes Best for One large model, long context, streaming Many sub-40B specialists, short requests See more about SIE vs LLM. When should you still choose vLLM? When you have one very large generative model that needs a multi-GPU node with long-context streaming. It works wonders there. vLLM should not act as a multi-model controller, and that’s not its intended purpose. Mistake 4: Self-Hosting at the Wrong Time Self-hosting isn’t the default. It actually has a break-even point where it starts to make sense from a cost perspective. A hosted API charges for every token. A GPU, on the other hand, comes with a fixed hourly cost, and it becomes cheaper once you keep it busy enough. When renting is still the right choice: Below the break-even point, a hosted API is indeed the correct answer, not a compromise. Some examples include low or uneven volume, where a dedicated GPU would sit idle; a total inference bill that’s still under a few thousand dollars a month; or a task that genuinely needs the strongest frontier model. As you can see, you need to calculate your volume before you build the cluster. Self-host the steady, high-volume, latency-sensitive parts, and rent for uneven traffic and the hardest reasoning. Superlinked has a cost model and a token-cost breakdown if you want to check where you stand using your own numbers. The Fix So, what can you do to make sure you don’t make all four mistakes at once? Run your small-model layer from one place instead of assembling one runtime per model and your own scheduler. That’s what Superlinked Inference Engine (SIE) provides. It’s an open-source inference server that runs encoders, rerankers, extractors, and small generation models on your own GPU, from a laptop to a production cluster. Skip the Plumbing The value here doesn’t depend on a single product. Pack many models onto shared GPUs, own the embedding and reranking layer instead of renting it in the request path, check the break-even point before you buy hardware, and adapt small models when you can. A managed layer is worth it for the operations you’d otherwise have to build by hand, such as the gateway, shared queue, autoscaling from zero, and multi-model lifecycle. That’s exactly what SIE provides for the sub-40B range, with the same engine and model identifiers running locally. ★ Star SIE on GitHub · Run the quickstart · Browse the model catalog