Your Load Test Won't Find These Four Gaps. Learnings from OpenAI's Habitat

Four failure modes from OpenAI's Habitat post — and why you'll hit three of them at a thousandth of the scale. TL;DR OpenAI published an engineering post on Habitat, the storage platform behind ChatGPT: 70M+ requests/sec, 500+ PB, ~40 regions, 10× growth year-over-year for three years. It's worth reading in full. What struck me isn't the scale. It's that none of the four incidents they describe were capacity problems. Nobody ran out of CPU, memory, disk, or bandwidth. Every one was a coordination effect that appears when you scale out — and three of the four trace back to a library default that is perfectly correct on a small system. A response sitting ready but unparsed because the thread was busy — invisible on every database dashboard. A feature-flag SDK polling every 60 seconds with no jitter, stalling entire pods in lockstep. A connection pool's LIFO default creating a self-reinforcing outage that survived removal of its own trigger. 18 open connections to serve 6 concurrent requests, because process count multiplies connections. None of these need 70M requests/sec to bite you. They need many processes, which you probably already have. What Habitat is, briefly Habitat is OpenAI's online storage platform — the thing that answers "load this user's settings" and "fetch this conversation" before any product can render. It sits between every OpenAI product and the underlying stores (Azure Cosmos DB, caches, blob storage), and owns routing, authorization, encryption, caching, data residency, multi-tenancy, and rate limiting. It started at DevDay 2023 as a small Python client library talking to a single database. Today it serves 500+ petabytes. The claim I find most credible in the post is that the hard part wasn't the absolute scale — it was that the scale kept moving. Most infrastructure is built for 10× and expected to hold a few years. Habitat got 10× per year, three years running. Everything below is theirs, not mine. I'm interested in which parts generalize. The architectural move: a library is a deploy you can't control A shared library is a deploy you can't control By mid-2025 Habitat had outgrown being a client-side library, and the reason is a deployment problem rather than a technical one.To reduce blast radius they wanted to shard critical data across regionally distributed Cosmos accounts. That needed new routing logic in the client. So: add it behind a feature flag, spend days rolling it out across dozens of services, spend days more adding shadowing to verify the sharding was correct, spend days more fixing a bug the shadowing found. Then, at the finish line, one team rolled back their service for unrelated reasons — to a version with the old buggy client — and caused precisely the outage the whole project existed to prevent. That's the clearest argument for a storage service I've read. Not performance, not abstraction. Simply: if your logic ships inside other people's binaries, you do not control when it changes, and you cannot roll it back. The secondary benefit turns out to be the one that matters long-term. A single service is a single chokepoint for access control, audit logging, and limiting who can touch the underlying store. You can't enforce a data-access policy in a library that any client can pin to an old version. Bug 1: the database was fast, the user still waited Your database was fast. The user still waited. Habitat stayed on Python deliberately — the post calls it "a strategic incursion of technical debt," with an explicit bet that their own coding models would make the eventual rewrite cheap. Python peaked at 20M+ requests/sec. The dominant problem wasn't throughput. It was tail latency, and specifically this: asyncio gives you concurrency, not CPU parallelism. Habitat does plenty of CPU work — routing, compression, encryption, checksumming, hedging, shadowing — and all of it contends for one thread. In traces of their slowest requests, storage had already responded quickly. The request was stalled waiting for its coroutine to be rescheduled so it could parse a response that was already sitting in memory. Their measured scheduling jitter reached hundreds of milliseconds, and in edge cases seconds. The thing to sit with: in that failure, every downstream metric looks healthy. Your database dashboard shows no change. The latency is real, user-visible, and invisible where you're looking. Their measurement trick is the most portable idea in the post, and it's four lines of code: schedule a no-op task at a fixed interval and record actual-minus-expected start time. That delta is your event-loop delay. This generalizes past Python — it's goroutine scheduling latency in Go, GC pause time on the JVM, the same event loop in Node. Whatever your runtime, something between "response arrived" and "your code runs" can queue, and most teams don't measure it. Their fix was counterintuitive: run fewer concurrent requests per process and scale out the process count instead. Which sets up the next two bugs. Bug 2: every worker doing the same thing at the same moment This one is almost funny, and it's the most likely of the four to be lurking in your service right now. Their feature-flag SDK was on defaults: poll for config every 60 seconds, no jitter, and the config payload contained every production rule for every service. Separately, someone had decided to run 8 Python processes per pod for better CPU utilization. Combine those and once a minute, every worker on every pod simultaneously stopped serving requests to parse a large JSON blob. Two independently reasonable decisions — a polling default and a process-count default — multiplied into a synchronized fleet-wide stall. Neither is visible in code review of the other. This is what I mean by coordination rather than capacity: no resource was exhausted, the work was simply all scheduled at the same instant. The fix was the obvious one once CPU profiling found it: smaller config, longer interval, add jitter. But notice that the bug scales with process count, not with traffic. A hundred-QPS service with the same defaults has the same stall. Bug 3: LIFO ate the service How a connection pool default eats a service This is the best story in the post. Symptom: a client burst overloaded part of the service. They stopped the client. The degradation didn't stop. Affected processes kept getting more traffic, not less, and only a restart cleared it. Cause: Python's aiohttp TCPConnector defaults to LIFO connection reuse — the most recently returned connection is handed to the next request. That's a sensible default. It lets surge connections idle out instead of being kept alive forever. But trace what it does under load. A struggling pod responds slower, so its connection returns to the pool later, so LIFO — which prefers the most recently returned — picks it first. The pod that's already behind gets the next request. And the next. They name it correctly: a metastable failure. The system has two stable states, and once the burst kicks it into the bad one, removing the burst doesn't kick it back. Switching to FIFO broke the loop and, as a bonus, reduced steady-state variance too. What makes this worth generalizing is that the default isn't wrong. It's optimal for connection lifetime management and pathological for load distribution, and nothing in the API tells you that you're making that trade. Go look up your connection pool's reuse order. Most engineers I know — myself included, before reading this — have never checked. Bug 4: 18 connections to serve 6 requests Same traffic. 18 connections, or 1. Scaling out to fix Bug 1 makes this one worse. Each process keeps its own connection pool, sized for its own peak. Multiply by process count and you get a connection count that has nothing to do with your actual concurrency. Their figures: six worker pods handling six concurrent requests hold 18 open connections — six busy, twelve idle. Put a shared proxy in front and it's six. Have that proxy upgrade HTTP/1 to HTTP/2 so requests multiplex as streams, and it's one. The consequences aren't about bandwidth. Connection count saturates resources nobody is watching: a routine daily deploy burns CPU cycling connections, a leak saturates the NAT gateway, downstream services get a thundering herd that their throughput-based capacity planning never predicted. The proxy earns its place for a second reason: it's somewhere to put rate limits and circuit breakers. Those are much less effective implemented independently inside hundreds of processes, each with a local view of a global problem. The best idea in the post: Habitat does less The section I'd steal outright is the one explaining why Habitat exposes a deliberately weak API. No arbitrary SQL. A constrained NoSQL interface over client-defined objects and edges, explicitly inspired by TAO. Their reasoning: it is cheap and easy to write SQL queries that are expensive and hard to run They call it cost imbalance, and they learned it the hard way. On Postgres, reviewing every query and schema change for well-behavedness worked until the team grew, at which point a single expensive query on a hot path taking out the database became routine. So Habitat optimizes for predictable, constant-work requests, and makes expensive operations obvious on the client side. Objects colocate with their edges in one partition, but no effort goes into colocating the objects an edge points to — so the model partitions cleanly for horizontal scale and graph traversal is deliberately inefficient, potentially crossing database accounts in different regions. Teams who genuinely need complex queries get an escape hatch: change data capture into a separate analytical store that they provision and scale themselves. That last part is the discipline most teams skip. It's easy to say "we only support simple queries." It's harder to also build the pressure valve, and without one your constrained API just becomes a thing people route around. There's a connection here to rate limiting that I don't think is accidental. You cannot meter what you cannot price, and you cannot price a request whose cost you don't know until it finishes. Habitat's answer is to refuse to accept unpredictable requests at all — which makes isolation, load balancing, and capacity planning tractable downstream. Constraining the API isn't a limitation of the platform. It's the mechanism that makes everything else in the platform possible. What the post doesn't say Three things I noticed by their absence: No latency numbers. For a post substantially about tail latency, there isn't a single millisecond figure. We're told scheduling jitter reached "hundreds of milliseconds" and that the Rust rewrite has "significantly lower" latencies. Significantly lower than what? The headline claim is also marketing. Two engineers plus Codex and GPT-5.5 rewriting the second-largest service by core count, in one quarter, now serving 95% of production at 6× the CPU efficiency and 15× the memory efficiency — that's genuinely impressive if it holds. It is also a very effective advertisement for the products that did it. No detail on timeline, test strategy, or how the cutover was validated. I'd want that detail before drawing conclusions about my own migrations. Nanobase appears in their architecture diagram and is never mentioned again. Presumably part two. Five things to check in your own service None of these require your scale to change: Do you measure your runtime's scheduling delay? Event loop lag, goroutine latency, GC pause. Not CPU utilization — the gap between "ready to run" and "running." Does any periodic task in your fleet poll without jitter? Config refresh, health checks, metric flush, cache warming. Any of them, across N processes, is a synchronized stall. What's your connection pool's reuse order? If you can't answer, you're running whatever the library chose. How many connections does your fleet hold per unit of concurrent work? If that ratio is well above 1, you're paying for idle connections and exposing yourself to churn. Can a client write a request whose cost you can't predict? If yes, every capacity and isolation decision downstream becomes guesswork. Close The reason this post is worth your time isn't the 70 million requests per second. It's that at that scale, ordinary defaults stop being ordinary — and reading about it is much cheaper than discovering it. Scaling out doesn't divide your problems across more machines. It multiplies the number of things that can accidentally happen at the same time. Source: OpenAI's engineering post, "Rapidly scaling online storage to serve over 1 billion ChatGPT users" (September 2026), by Jon Lee, Chaomin Yu, and Ben Ries. Part two is promised on the storage layer and their Cosmos DB work. All incidents described above are theirs; the framing and the checklist are mine.