Small Specialized Models Are Eating the AI Stack (While Everyone Watches Frontier LLMs)

Every few weeks or so, updates light up for a new frontier model. New leaderboards, new benchmarks, screenshots, promises of how this one will be the one that “changes everything”… By now, I’d say we’ve all grown accustomed to it. But under the hood, most agents rarely route every task to a frontier model. Reranking, embeddings, OCR, and entity extraction generally go to smaller models. In fact, the majority of the models doing the heavy lifting inside an AI agent are not the giant LLMs everyone’s watching. They are small, specialized, and open. Let me show you below where the work really goes and why these small models are good enough. Where Does an Agent Spend Inference? If I asked you to picture an "AI agent", you’d probably imagine an LLM doing the thinking. And that’s fair, but what is it that actually happens on every query? Generation happens only once or twice per turn. Yet, retrieval, reranking, and extraction happen constantly. As Superlinked put it in their piece on small models for agent tasks: The work that scales with agent usage is rarely generation. It is the repetitive, high-volume inference: embedding every chunk, reranking every retrieval, extracting fields from every document. Those are exactly the jobs small specialized models do well. Here’s a rough anatomy of a single agent turn over a document-heavy workload: Task Who Does It How Often It Fires Understand and answer Frontier or mid-size LLM (generation) Once or twice Embed the query and new chunks Dense encoder (for example stella, bge-m3) Every chunk, every turn Rerank candidates before context Cross-encoder reranker Every retrieval Pull entities, dates, amounts Extraction model (for example GLiNER) Every document Read a scan or page image Optical Character Recognition (OCR) model Every image or scan With this, it’s easy to notice a pattern. The expensive model runs the least, and that’s on purpose. The small models, on the other hand, run in a loop. Their volume is what appears on the invoices and has the biggest impact on p99 latency. TL;DR: In an agent, generation is just the tip of the iceberg. Embedding, reranking, and extraction are what lies below the surface. Small models handle most of that work. Are Small Models Any Good for This? A “small model” might sound like a “worse model” to you, and for hard reasoning, it might be. But when it comes to the specific jobs mentioned above, such as embedding, reranking, extracting, and OCR, that’s where they typically shine. On embeddings, Superlinked benchmarked self-hosted open models against hosted frontier models from Voyage, OpenAI, and Cohere across eight standard Massive Text Embedding Benchmark (MTEB) retrieval tasks. The head-to-head results put the open models at a mean 0.600 normalized Discounted Cumulative Gain (nDCG) versus 0.615 for hosted frontier models. That’s roughly 97.5% of the quality, while winning outright on two of the eight tasks. On the cost side, however, it stops being so close. Embedding a billion-token corpus cost just $10 to $17 for self-hosted models. The hosted APIs cost $120 to $130 for the same corpus. The story with generation is similar and a bit surprising. Superlinked’s CEO Daniel Svonava has cited an Artificial Analysis Intelligence Index snapshot, where Qwen3.6-27B scored about 46 against GPT-5.1's 48 (roughly 96% of the score) despite being released about five months apart. ⚠️ These numbers change over time and vary by reasoning mode, so treat them as a cited snapshot. Make sure to check the live comparison for the latest results. Now back to money. Intercom has reported cutting roughly $250,000 a month by replacing a hosted GPT call with a fine-tuned 14B Qwen model for a specific Fin AI pipeline task.Also, open weights aren’t frozen. You can adapt them cheaply. Where the frontier model still wins: On the hardest novel reasoning. Job Small Specialized Model Frontier LLM / hosted API Embeddings for search and Retrieval-Augmented Generation (RAG) Best for: cost, speed, self-hosting Best for: zero-ops, spiky low volume Reranking Best for: precision per dollar Not worth an API call Entity and field extraction Best for: high-volume, schema-shaped output Best for: fuzzy, open-ended reads OCR and document vision Best for: predictable batch cost Best for: one-off or rare formats Hardest novel reasoning Not the right choice for this Best for: this, genuinely What’s the Catch? If small models are this good and this cheap, then why’s everyone still funneling the small stuff through hosted APIs? Because serving lots of small models is a fundamentally different infrastructure problem than serving one big model. Not to mention the fact that most tooling was built for the latter. Here’s how the SIE docs describe this: Inference tools are designed for one large model spread across many GPUs. Small-model inference is the opposite. You run many models (encoders, rerankers, extractors) on a single GPU and need fast switching between them. If you get this wrong, you’ll end up in a classic trap. Every model has its own deployment and dedicated GPU pool. Five models = five pools, each provisioned for peak load and idle the rest of the time. In other words, you may end up renting five GPUs to do the work that one could handle, while paying for the other four to just sit idle. Meet SIE SIE (Superlinked Inference Engine) is an open-source inference server for AI models. Instead of running one service per model, it packs multiple onto each GPU and puts them behind a single API. It ships with 100+ models out of the box, runs the same Docker image from a laptop to a production Kubernetes cluster, and is released under the Apache 2.0 license. The whole surface comes down to four primitives: Encode turns text or images into vectors for semantic search and RAG. Score reranks query and document pairs for higher-precision retrieval. Extract pulls entities and structured data out of unstructured text. Generate runs text generation on small open LLMs you host yourself. The multi-model trick underneath is where the interesting, geeky stuff happens. SIE lazy-loads models onto shared GPUs and evicts the least-recently-used one when memory fills up. An L4 with 24 GB keeps two to three standard models hot while the rest of the catalog stays available on demand. Here is how it stacks up against the two tools devs usually reach for: SIE TEI (Hugging Face) OpenAI API Self-hosted Yes Yes No Multi-model on one GPU Yes No (one model per server) Not applicable Encode + score + extract + generate Yes Encode only Encode + generate only 100+ models Yes Varies Limited Per-token cost None None 💸️ When Frontier LLMs and Hosted APIs Still Make Sense It would be wrong of me to tell you to self-host everything. So, here’s where frontier LLMs and hosted APIs still make financial and development sense: Hosted APIs win on zero-ops convenience and on spiky, low-volume traffic. Frontier models win on the hardest reasoning. Superlinked's break-even guidance is very clear: Below roughly 5 million requests a month, a dedicated GPU just sits idle. If your whole inference bill is still under about $2K to $3K a month, running your own GPU infrastructure isn’t worth your time yet. Skip the Wiring If you’ve come this far, you’re already aware of the trade-offs. The small models are ready, the hardware is a single GPU, and the only realistic friction is the serving layer underneath. You can absolutely build that layer yourself. Teams have been known to do it. But if you’d rather not hand-roll routing, shared queues, autoscaling, and multi-model lifecycle, that’s exactly what SIE is for. And it’s open source, which is lovely. If you want someone else to run the cluster, Superlinked offers that too. Either way, the <10 minute quickstart is probably the cheapest way to find out whether your small-model bill can put your EBITDA on better terms. Frequently Asked Questions (FAQ) Are small models good enough to replace frontier LLMs? Yes, when it comes to high-volume jobs an agent repeats (embeddings, reranking, extraction, or OCR). What counts as a "small" model here? Roughly the 100M to 1B parameter range for encoders, rerankers, and extractors, and up to sub-40B for generation models that fit on a single GPU from two to four hardware generations ago. See Superlinked's self-hosting guide for the sizing details. Why not just run one inference server per model? Because pinning one model per GPU leaves you at around 3% utilization across a five-model pipeline. SIE packs several models onto a shared GPU with least-recently-used eviction, so you can rent far fewer units. Does SIE run generation too, or only embeddings? It does both. SIE exposes four primitives: encode, score, extract, and generate. Generate runs small open LLMs you host yourself. It’s not aimed at replacing a hosted frontier model for the hardest reasoning. When should I not self-host? When your traffic is spiky or under about 5 million requests a month, when your total inference bill is still under about $2K to $3K a month, or when you truly need the absolute frontier model. In those cases, a hosted API is the simpler and cheaper option.