FastAPI Middleware Error Recovery: Preventing One Broken Tenant from Taking Down Your Entire SaaS

FastAPI Middleware Error Recovery: Preventing One Broken Tenant from Taking Down Your Entire SaaS I learned this lesson the hard way at 2 AM when a customer's misconfigured webhook integration sent 50,000 malformed requests per second to our shared FastAPI origin. One tenant's mistake nearly took down nine others. The middleware caught it, logged it, and the system kept breathing. That's when I realized: middleware isn't about prettifying requests—it's your firewall against cascade failures. Most FastAPI developers treat middleware as a transformation layer. Request comes in, you extract headers, add context, pass it along. But in multi-tenant systems, middleware is actually your last defense against one tenant's chaos rippling through your entire infrastructure. I'm going to show you exactly how to weaponize it. The Problem: Why Standard Error Handling Fails in Multi-Tenant Systems When a request handler throws an exception in a single-tenant app, FastAPI's built-in exception handlers catch it and return a 500. Users see an error. Life goes on. But in multi-tenant systems: One tenant's bad data causes a database deadlock → affects all tenants querying that table A malformed request body exhausts memory → slows down request processing for everyone An uncaught exception in business logic → leaves the connection pool in an unknown state Logging doesn't include tenant context → you can't debug whose request caused the crash I've watched this happen. The culprit wasn't even running expensive queries—it was sending requests so malformed that the validation layer itself crashed. Without tenant-aware middleware, you're blind. The Solution: Defensive Middleware Layers The pattern I use now: build a middleware stack that catches exceptions at the boundary, logs tenant context immediately, and returns safe responses without exposing internal state. Here's the core structure: from fastapi import FastAPI, Request, status from fastapi.responses import JSONResponse from contextlib import asynccontextmanager import logging import time import uuid from typing import Callable app = FastAPI() logger = logging.getLogger(__name__) class TenantContextMiddleware: """Extract and validate tenant from request BEFORE it hits handlers.""" def __init__(self, app): self.app = app async def __call__(self, request: Request, call_next: Callable): # Extract tenant ID from subdomain or header tenant_id = request.headers.get("X-Tenant-ID") or self._extract_from_subdomain(request.url.hostname) request.state.tenant_id = tenant_id request.state.request_id = str(uuid.uuid4()) request.state.start_time = time.time() response = await call_next(request) return response def _extract_from_subdomain(self, hostname: str) -> str | None: if hostname and "." in hostname: parts = hostname.split(".") if parts[0] not in ("www", "api", "app"): return parts[0] return None class TenantErrorBoundaryMiddleware: """Catch ALL exceptions, log tenant context, return graceful 5xx.""" def __init__(self, app): self.app = app async def __call__(self, request: Request, call_next: Callable): try: response = await call_next(request) # Check for 5xx status codes and log them if 500 1000: logger.warning( f"Tenant {tenant_id} exceeded rate limit", extra={"tenant_id": tenant_id, "count": current_count} ) return JSONResponse( status_code=status.HTTP_429_TOO_MANY_REQUESTS, content={ "error": "Rate limit exceeded", "retry_after": 60, "request_id": request.state.request_id, } ) response = await call_next(request) response.headers["X-RateLimit-Remaining"] = str(1000 - current_count) return response app.add_middleware(TenantQuotaMiddleware) app.add_middleware(TenantErrorBoundaryMiddleware) app.add_middleware(TenantContextMiddleware) Enter fullscreen mode Exit fullscreen mode The order matters. Tenant context → quota enforcement → error boundary. This ensures you catch rate-limited requests before they even enter your business logic. What I Missed (and learned from) Gotcha #1: Middleware order is execution order in reverse. I registered them in logical order once and spent an hour wondering why tenant context wasn't available in the quota middleware. FastAPI executes middleware in reverse registration order—the last one added runs first. Gotcha #2: Async context managers aren't your friend here. I tried using @asynccontextmanager for cleanup logic (closing connections, rolling back transactions). It worked locally but failed under high load because the context wasn't properly propagated. Stick to explicit try-finally blocks in middleware. Gotcha #3: Logging context gets lost across async boundaries. I use structlog with context vars now instead of passing state through request objects for log enrichment: from contextvars import ContextVar tenant_context: ContextVar[str] = ContextVar("tenant_id", default=None) class TenantContextMiddleware: async def __call__(self, request: Request, call_next: Callable): tenant_id = request.headers.get("X-Tenant-ID") token = tenant_context.set(tenant_id) try: return await call_next(request) finally: tenant_context.reset(token) Enter fullscreen mode Exit fullscreen mode Now every log across the async chain includes tenant context automatically. The Real Win Here's what actually matters: that 2 AM incident? With this middleware stack, the bad tenant got rate-limited in 30 seconds. Their requests returned 429 responses. The database stayed responsive. Other tenants never knew something happened. I got a Slack alert with request_id and tenant_id already in the context, debugged the issue in 10 minutes, and went back to sleep. That's what defensive middleware buys you in production.