The Token Price Fallacy: Why Your Agentic AI Bill Keeps Growing While Unit Costs Collapse

Cost per token is falling fast. Cost per completed task isn't, and almost nobody is measuring the difference. The cost to query a model at GPT-3.5-level performance fell 280 times between November 2022 and October 2024, from $20 to $0.07 per million tokens, according to Stanford HAI's 2025 AI Index Report. By every unit-price chart, inference should be close to free by now. McKinsey's July 2026 research on agentic AI spend found the opposite in practice: 93% of enterprises running agentic AI are exceeding their AI budgets, and one in five has constrained its use due to cost. Enterprise LLM spending tripled over the 12 months to the end of 2025, while unit prices kept dropping over the same period. I research token optimization and CO2-aware inference alongside my day job leading Salesforce CPQ and CLM work at one of the largest telecom companies because I keep running into the same gap between the pricing charts vendors show and the bills enterprises actually get. The chart measures the wrong unit The cost per token is a real number, and it's really falling fast. Epoch AI's research on inference economics found the pattern holding into 2026: at a fixed capability level, cost per token drops roughly 5 to 10 times a year. One example from their February 2026 analysis: matching the same score on the FrontierMath benchmark took 43 million output tokens with an April 2025 model and 5 million tokens with a December 2025 model, a threefold cost drop in 8 months for the same result. None of that shows up as savings if the number of tokens per task is also rising, and for agentic systems, it is. Gartner's March 2026 forecast puts agentic workloads at 5 to 30 times the tokens per task of a standard chatbot exchange because an agent doesn't answer just once. It plans, calls tools, checks its own output, and sometimes retries the whole chain when a step fails validation. Cost per successful task is the number that matters, and it's the number almost nobody is measuring. It's the product of at least four variables, and treating any one of them as the whole story is how budgets blow past forecast. The four variables Tokens per task Not tokens per call. A single agentic task might involve a planning step, three tool calls, two self-checks, and a final synthesis. Gartner's 5-to-30 times multiplier is an average across task types; complex multi-step workflows sit at the high end of that range. Measure your own multiplier by task category before you forecast cost from a per-token price sheet. Retry and refinement overhead McKinsey's July 2026 research found that about 60% of an agentic task's cost is tied to refining answers: retries, re-prompts, and validation loops that fire when the first pass doesn't meet a quality bar. That work makes agentic output reliable enough to ship, and it's a real, necessary cost. It also means the sticker price on a model card describes a fraction of what a completed task actually costs. Model routing If cost per token at fixed capability keeps falling 5 to 10 times a year, the model that was right-sized for a task in January is very likely oversized by December. Most enterprise deployments I've seen don't re-benchmark task-to-model assignment on any regular cadence. They pick a model at launch and leave it running against workloads it's now overpowered for, because nobody owns the job of checking. Energy and carbon per query This is the variable with the least agreement, and it's worth being honest about the range rather than picking one number and presenting it as settled. A 2025 peer-reviewed estimate (Jegham et al.) put GPT-4o-class queries at 0.42 to 1.79 watt-hours and under 0.3 grams of CO2 equivalent, while a reasoning-heavy model like DeepSeek-R1 ran 24 to 34 watt-hours and over 14 grams of CO2e for a comparable task, roughly a 20 to 50 times spread depending on how much the model reasons before answering. Google's own August 2025 disclosure reported its median Gemini text query at 0.24 watt-hours and 0.03 grams of CO2e, a 33- to 44-fold drop from its own estimate a year earlier as infrastructure and model efficiency improved. The spread across sources is close to two orders of magnitude. Anyone quoting a single "CO2 per AI query" number without naming the model, the reasoning depth, and the measurement methodology is quoting a number that isn't comparable to anyone else's. What this costs enterprises that don't measure it Gartner's January 2026 research on GenAI project failure found that at least half of GenAI projects get abandoned after the proof-of-concept stage, citing poor data quality, inadequate risk controls, and escalating costs as drivers. What looks manageable in a pilot can turn into what one Gartner analyst calls a budget black hole once token usage scales across real users. The infrastructure spending data backs this up from the supply side. Gartner's August 2026 forecast on AI-optimized cloud infrastructure found inference spend crossed training spend for the first time this year, at 55% ($23.3 billion) of a $42 billion market versus 45% ($19 billion) for training, and projects that split to widen to 59% inference on $66 billion in 2027. The cost center inside enterprise AI has permanently shifted from building models to running them, and most cost governance processes haven't caught up. A measurement framework, not a pricing chart If you're accountable for an agentic AI budget, track four numbers per task category, not one blended average across everything running in production: Average tokens consumed per completed task, broken out by planning, tool calls, and self-correction steps. Percentage of tasks that require at least one retry or refinement pass, and the token cost of that pass. Days since the model assigned to this task category was last benchmarked against a smaller or newer alternative. An energy or carbon estimate for the task, sourced from the model provider's own disclosure where available, is treated as directional rather than precise, given the width of the published range. None of these numbers come from a vendor's per-token price sheet. All four come from instrumenting your own production traffic, which is more work than reading a pricing page and less work than explaining a budget overrun after the fact. The 280-fold price drop is real. So is the tripling of enterprise LLM spend in the same period. Both are true at once because they're measuring different things, and until cost-per-successful-task replaces cost-per-token as the metric teams report on, that gap will keep showing up as a budget surprise rather than a forecast.