The 3× Token Bill We Didn’t See Coming

I was going through our agentic application dashboard and noticed a spike in last week’s LLM usage, with traffic sitting at exactly the same level it had been for weeks. Initially we assumed it was a logging bug but it caught a real issue. It lined up almost exactly with an architectural change we’d shipped a few weeks earlier: we’d moved a chunk of our pipeline from a single-agent setup to a multi-agent one. Different agents handling different parts of a task, coordinated through LangGraph, with a supervisor node deciding what happens next. On paper this was a clean win accomplishing better task decomposition, each agent doing one thing well instead of one model trying to do everything inside a single sprawling prompt. You’d expect usage to go up somewhat, more calls per task is the cost of that architecture, and nobody was pretending otherwise. But to our surprise it had roughly tripled for tasks that functionally hadn’t changed at all. What’s in this article The intuition that stopped working Chasing the wrong suspect The fix that looked right and wasn’t enough What actually worked What the numbers looked like once we stopped guessing What I’d still reconsider The intuition that stopped working If you’ve only ever run a single LLM call per task, your mental model of cost is roughly linear, longer input, longer output, more tokens, more spend. You can eyeball it and be close enough. That intuition falls apart the moment you introduce orchestration, and it falls apart quietly, which is worse than falling apart loudly. A supervisor agent now makes a decision about what happens next, and that decision itself costs tokens. Each sub-agent carries its own system prompt, its own tool schemas, often its own copy of context the previous agent already had. None of this shows up as “more work being done”, from the user’s side the task is still the same task. It shows up as more machinery wrapped around the task, and machinery isn’t free just because it’s invisible to whoever’s using the product. We hadn’t budgeted for it. Our architecture reviews had focused on decomposition quality and correctness, did each agent do its job well, did the handoffs make sense, and not once, as far as I can remember, on what the token graph would actually look like once it was live and taking real traffic. That’s on us, and it’s probably on most teams making this move for the first time. Chasing the wrong suspect My first assumption was fan-out. Somewhere, the supervisor was spawning more sub-agent calls than the task genuinely needed, and trimming the branching logic would bring the number back down. It felt like the obvious place to look, more agents, more calls, simple arithmetic. So I spent the better part of a day instrumenting call counts per supervisor decision, expecting to catch some node calling three agents when one would have done the job. I found almost nothing. The fan-out was doing roughly what it was meant to do, branch for branch. That was frustrating in the specific way that being wrong about an obvious answer is frustrating. You’ve burned a day, and you’re back where you started with one hypothesis eliminated and no replacement. The replacement turned up almost by accident, while I was reading through raw call logs looking for anything unusual rather than testing a theory. One agent’s output would fail a validation step, a malformed tool call, a schema mismatch, occasionally a model deciding to explain itself in prose instead of just calling the tool and the retry wrapper would silently re-invoke that agent. Silently, because it completed successfully on the retry and nothing downstream ever saw a failure. Which sounds fine, until you notice what re-invoking that agent actually required: rebuilding its upstream context, which in some paths meant re-running a sub-agent that had already produced a perfectly good result, purely to reconstruct the input the failing agent needed. One validation failure, three layers deep in the graph, could trigger a cascade that re-ran work that had never been broken. Nothing about this threw an error a human would see. The pipeline completed and the answer that came back was correct. The retry logic was doing precisely what it had been told to do, retry on failure, it simply had no concept of how expensive the thing it was retrying actually was, or whether re-running already-correct upstream work was necessary at all. If you’re building retry logic for an agent graph rather than a single call, this is the part worth sitting with: a retry decorator that behaves perfectly well wrapped around one LLM call becomes a cost amplifier the moment it sits inside a graph with shared upstream state. The fix that looked right and wasn’t enough The obvious response to “retries are expensive” is to cap retry counts and add exponential backoff. We tried it but it only helped a little. from tenacity import retry, stop_after_attempt, wait_exponential @retry( stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10), reraise=True, ) def call_agent(agent, payload): return agent.invoke(payload) It’s not the wrong thing to have and you should have retry caps regardless of anything else in this article but it didn’t touch the actual problem. Every retry, capped or not, was still paying full price on the most capable, most expensive model available in the graph, regardless of what had actually gone wrong. A malformed tool call from a summarisation step doesn’t need the same firepower to fix as a genuinely hard reasoning failure. Treating every retry as equally deserving of your strongest model is the mistake, and it’s the one I see recommended most often: add backoff, add a retry limit, move on, as if the cost per retry weren’t the variable actually worth controlling. Capping how often you’re wrong doesn’t help you if you’re still paying premium prices to be wrong. What actually worked The real fix was three changes, and they only really worked once we’d made all three. Doing just one of them left an obvious gap the others would have closed. Routing by task complexity – Sub-agents got assigned a model based on what the step genuinely required, rather than one model used everywhere by default. Summarisation, formatting, routine tool-call construction, the mechanical steps, not the reasoning-heavy ones moved to a smaller, cheaper model. Steps that needed actual multi-step reasoning or judgement calls, the kind of thing that had been causing the interesting failures in the first place, stayed on the stronger model. It’s a simple table, and the reason it hadn’t existed from the start says more about our priorities. The original goal had been correctness-per-call and correctness-per-call quietly biases you towards “just use the best model everywhere and worry about cost later.” I don’t think that’s a defensible default once you’re past a prototype. If you can’t articulate why a given step needs your most expensive model, it probably doesn’t, and writing the routing table is the exercise that forces you to actually answer that question instead of assuming it. Context trimming between handoffs – This one’s easier to skip because it’s less visible than model choice, and we very nearly did skip it. Every agent in the graph was inheriting the full accumulated context from everything upstream of it, by default, because that’s what the framework gives you unless you actively decide otherwise. A formatting agent at the very end of the chain was receiving the entire reasoning trace of three earlier agents, most of which it had no use for whatsoever. def trim_handoff_context(state, needed_fields): return { field: state[field] for field in needed_fields if field in state } # formatting agent only needs the final structured result, # not the full upstream reasoning trace handoff = trim_handoff_context( graph_state, needed_fields=["final_result", "output_schema"], ) We trimmed handoffs down to what the receiving agent actually needed: final outputs and the specific fields relevant to its task, not the full upstream conversation. It doesn’t produce one dramatic before-and-after number but what it produces is a steady reduction across every single call in the graph, because you’ve stopped paying input-token cost for context that was never going to be read by anything downstream. Parallel execution – We’d originally parallelised independent sub-agent branches purely to cut latency. Agents doing separate jobs, pulling different data sources, running independent checks don’t need to run sequentially just because a sequential graph is easier to draw on a whiteboard. import asyncio async def run_independent_branches(agents, payload): results = await asyncio.gather( *(agent.ainvoke(payload) for agent in agents), return_exceptions=False, ) return results What I hadn’t expected going in was that parallel execution also shrank the retry blast radius, almost as a side effect. Sequential execution meant a downstream failure could trigger re-running an entire upstream chain to rebuild shared state. Independent parallel branches meant a failure in one branch had nothing to drag down with it because there was no shared state to rebuild in the first place. Latency was the reason we did this but in hindsight, the reduced retry cascading is the part that mattered more. What the numbers looked like once we stopped guessing I want to be upfront that these are rough figures, not precision-instrumented ones, because we hadn’t wired up per-step cost attribution before any of this started which is itself the lesson, build that first, not after you’ve already got a problem to diagnose. With that caveat on the table: token usage per model dropped by roughly 40% once retries stopped silently re-running already-correct upstream work, and end-to-end latency dropped somewhere in the 45–55% range from the combination of task-based routing and parallel execution. The retry fix and the routing fix landed close together in time, so I can’t cleanly separate how much each one contributed on its own. That’s a genuine limitation of how we measured this, not a hedge for the sake of sounding careful. What I’d still reconsider Complexity-based routing works, but it’s a table someone has to maintain by hand, and it goes stale the moment you add a new agent type or a model gets deprecated out from under you. A lightweight classifier could predict task complexity and pick the model at runtime instead but that’s its own model call, with its own cost, sitting there guarding against a cost problem. Whether that trade is worth making depends entirely on how often your task mix actually shifts, and we haven’t run this long enough, on a wide enough variety of tasks, to know the answer for us yet. The biggest lesson wasn’t “use smaller models” or “cap retries.” It was that multi-agent systems have failure modes that don’t exist in single-agent pipelines. Once work is distributed across a graph, every retry, every context handoff, and every routing decision becomes part of your cost model. If you’re only measuring latency and correctness, you’ll discover those costs in production rather than during design.