Three Generations of Autoscaling — And Why Agentic Traffic Breaks All of Them

All images were created by the author using [Power Point / Copilot] In my organization I’ve worked as a backend engineer and architect. My main responsibility is ensuring the services we design meet their functional requirements, but also scale to millions of requests per minute, hold a 99.99% uptime, and stay cost-effective enough to keep OPEX in check. For most of my career, the traffic hitting those services followed the trend I could reason about — human-driven, forecastable, self-limiting. That’s changed. Agent traffic doesn’t behave that way, and the two dominant scaling models (on-demand and serverless) we’ve built both break under it. In this article, I walk through the mindset shift needed when scaling for agentic traffic — and the specific patterns worth considering. If you’ve been in AI engineering, you’ve watched this shift happen. The traffic hitting your endpoints, your model gateways don’t look the way traffic used to look, or the way human traffic used to behave. Agentic traffic comes in unpredictable bursts. It repeats itself and retries relentlessly. It drains your scaling costs significantly compared to human-shaped traffic. I want to walk through why that assumption is now broken, why the two dominant scaling models (on-demand and serverless) we’ve built inherit the problem, and what the solution to the problem should look like. The assumption underneath everything Every scaling framework we’ve built assumes traffic looks the way people generate it. Compare what’s on the left of this table with what’s now on the right DimensionHuman-driven trafficAgent-driven trafficShapeDiurnal curve with forecastable peaks. Tomorrow looks like today.No schedule. Bursts triggered by orchestration events, not clocks. The pattern itself shifts as agents and prompts change.Onset speedRamps over seconds to minutes; you can watch it build.Near-instantaneous. A parallel fan-out or a tight loop reaches full rate in milliseconds — faster than reactive scaling can respond.ConcurrencyIndependent users; the aggregate smooths out by the law of large numbers.Correlated fan-out from a single trigger. One orchestration spawns many synchronized calls. No statistical smoothing.RetriesBounded. People give up, refresh occasionally, back off out of frustration.Programmatic and relentless. Without an explicit retry budget, an agent turns one fault into a retry storm.Latency toleranceSub-second or the user abandons.Often tolerant of seconds to minutes — reasoning runs in the background. This slack is exploitable.Cost driverRequest count roughly tracks cost.Request count is decoupled from cost. One heavy reasoning chain can consume more compute than a thousand lightweight calls.Failure modeGraceful degradation — users drop off.Self-amplifying. Loops drain resources and, on serverless, bill you for every redundant call before any signal fires. Agentic traffic violates all seven of these assumptions at once. That’s why the answer isn’t a better version of either model we already have — it’s a fundamentally different place to put the intelligence. Let me show you what I mean by walking through how the discipline of scaling has evolved. Generation 1: Anticipation (on-demand instances) Earlier in my career, working on large-scale video streaming backends serving millions of concurrent viewers, capacity planning was a human exercise. When a major live event was about to kick off, we knew exactly when the spike was coming, roughly how steep it would be, and when it would flatten. The work was in the anticipation: pre-warming EC2 fleets days ahead, setting min/max autoscale bounds, staffing a war room through the event. The traffic was human-shaped. It had a curve, a peak, a tail. You could reason about it and prepare for it. I used to spend hours in war rooms. It used to start hours before the event, making sure the EC2 fleets were all pre-configured, instance types were updated, health checks were working fine, load balancers were all good, and the network was in good health. We used to constantly monitor the spikes in call volumes and the failures. There were instances where the demand anticipation did not fit well because of misconfigured client-side call volumes, which led us to change the auto-scaling group policy during the event and even absorb a brief period of failures. Once the event was over, we’d see the traffic fall, and then after the event we’d need to fall back to previous capacity that was ramped up — to save on OPEX. Generation 1’s core assumption: the spike is forecastable, so provision ahead of it. Generation 2: Reactive trust (serverless) Then we started building serverless, which changed the game — API Gateway, Lambda, Step Functions, and the provisioning conversation largely disappeared. You stopped pre-warming and started trusting the platform to react. That worked because traffic was still mostly human-driven: users open the app, navigate, interact, and close it. Demand was still predictable, and platform reaction time was fast enough because onset was gradual. Generation 2’s core assumption: you don’t need to anticipate, because the platform reacts faster than demand ramps. The pivot: why machine orchestration breaks both at once Non-deterministic machine orchestration — autonomous agents, multi-step tool-calling chains, retrieval loops — breaks both models simultaneously. It defeats Gen 1 because there is no schedule to anticipate. Agent traffic has no clock and you can’t pre-warm for a spike you can’t predict. It defeats Gen 2 because reactive scaling is a lagging signal. Agent traffic reaches full rate in milliseconds; by the time CPU-based autoscaling fires, you’re already degraded. Worse, serverless faithfully executes every redundant call in a runaway agent loop — and bills you for the dysfunction. To build and respond to agentic requests, the four-layer response below covers the key patterns. All images were created by the author using [Power Point / Copilot] Layer 1: Behavior-based scaling You need to stop scaling on CPU usage. It’s a lagging signal and by the time it crosses a threshold, an agent loop has already drained the pool or run up the bill for dysfunction. The signal you want to monitor is request velocity and shape. Near-identical requests from one caller are an indicator of an agent loop to be checked long before it shows up in aggregate CPU metrics. An agent (retry, misconfigured) can send hundreds of near-identical requests in seconds. By the time CPU picks up the signal the loop cycles may already have degraded performance or wasted real money on serving those duplicate calls. The pattern below uses request velocity + payload diversity to figure out whether caller X is in a loop, so you can quarantine them before they burn out CPU cycles. import time from collections import defaultdict, deque class AgentLoopDetector: """Flags runaway agent loops by request velocity and payload repetition, well before aggregate CPU reflects the load.""" def __init__(self, window_s=10, rate_threshold=50, diversity_threshold=0.2): self.window_s = window_s self.rate_threshold = rate_threshold self.diversity_threshold = diversity_threshold self.events = defaultdict(deque) # caller_id -> deque[(ts, payload_hash)] def is_looping(self, caller_id: str, payload_hash: str) -> bool: now = time.monotonic() q = self.events[caller_id] q.append((now, payload_hash)) while q and now - q[0][0] > self.window_s: q.popleft() rate = len(q) if rate QUEUE_HIGH_WATERMARK: # Backpressure: tell the caller to slow down instead of queueing infinitely return Response( status=429, headers={"Retry-After": backpressure_delay(depth)}, body="system saturated, retry later", ) job_id = queue.enqueue(request.payload, caller_id=request.caller_id) return Response(status=202, body={"job_id": job_id, "poll": f"/result/{job_id}"}) # Worker side: pull at a controlled rate; concurrency caps protect downstream def worker_loop(): for job in queue.consume(max_concurrency=200): result = process(job) results.put(job.id, result) Layer 4: Token-based admission control Instead of counting requests in aggregate, the intent is to shift the unit of admission from request count to resource cost. Don’t cap calls per minute; cap the compute a session can consume. A token bucket keyed on session — debited by actual tokens or compute used, not by call count — lets a heavy reasoning chain that consumes disproportionate compute be cut off, while lightweight callers pass freely. Below is an example of SessionTokenBucket, which implements per-session admission control by token cost, not request count. Each session gets its own bucket of capacity_tokens (default 100,000) that refills at refill_per_s (default 1,000 tokens/second). The admit() method estimates the cost of an incoming call and either debits the bucket if enough tokens are available or rejects the call. The core mechanism is time-based refill: when a session tries to admit, _tokens() computes how many tokens have accrued since its last activity, capped at the bucket size. This lets an idle session build up capacity, while an active session gets throttled proportional to its consumption. The usage is straightforward — if admit() returns False, respond with an HTTP 429 (Too Many Requests) telling the caller their session budget is exhausted. Lightweight callers keep passing through unaffected; a session running a heavy reasoning chain gets cut off before it drains resources everyone else needs. import time class SessionTokenBucket: """Admission by resource cost. Capacity and refill are in tokens (compute), not requests — so one heavy reasoning chain can be rejected while many light calls pass.""" def __init__(self, capacity_tokens=100_000, refill_per_s=1_000): self.capacity = capacity_tokens self.refill = refill_per_s self.state = {} # session_id -> [tokens_available, last_refill_ts] def _tokens(self, session_id): now = time.monotonic() avail, last = self.state.get(session_id, (self.capacity, now)) avail = min(self.capacity, avail + (now - last) * self.refill) self.state[session_id] = [avail, now] return avail def admit(self, session_id, est_tokens) -> bool: if self._tokens(session_id) = self.breaker_threshold: self.open_until = time.monotonic() + self.cooldown_s raise CircuitOpen("breaker tripped") delay = resp.headers.get("Retry-After") or (2 ** attempt + random.random()) time.sleep(float(delay)) # cooperate, don't hammer continue self.failures = 0 return resp raise RetryBudgetExhausted("stopped asking") # the client decides to stop Where this leaves us We spent Generation 1 with the load anticipating. We spent Generation 2 trusting the platform to react. Generation 3 asks something harder: build clients and infrastructure smart enough not to generate the load in the first place. Even with Generation 1 and Generation 2 for the deterministic, human-driven load, I’ve seen misbehaving clients that lead to issues. The client needs to be smart enough to understand the ask in the first place and respect all backpressure signals. If you’re designing an agent architecture today — orchestrating LLM calls, running retrieval pipelines, letting language models plan and act — build the retry budget, the circuit breaker, and the cooperative backpressure into the client from day one. Don’t leave it as an afterthought that surfaces when it starts costing you money. Again, the smartest valve isn’t at the pipe entrance. It’s at the source.