Designing Reliable LLM Agents With Deterministic Control Flow

Every engineer who's shipped an LLM agent past the demo stage hits the same wall. Fifty lines of Python, an API key, a system prompt, and two tools is enough to build something that looks autonomous and feels magical. Point that same architecture at tens of thousands of real users, and the magic turns into 2 a.m. pages. I call this the Production AI Wall. It's the moment a team realizes the property that makes LLMs impressive in a demo, their probabilistic, "creative" behavior, is the same property that makes them unreliable infrastructure. Traditional systems engineering is built on a simple contract: input *A* plus state *S* produces output *B*, every time. LLMs don't honor that contract. They sample a probability distribution over tokens. When we let an LLM freely control its own loop, deciding which tool to call, how to parse the response, when the task is "done", we're quietly swapping a deterministic finite state machine (FSM) for a high-variance statistical sampler, and then asking it to hit an SLA. The fix isn't a better prompt. It's inverting the design: stop relying on prompt engineering to enforce business logic, and instead wrap the non-deterministic model inside deterministic software boundaries. Three Ways Agentic Loops Actually Break Watching high-throughput agent pipelines fail in production, the failures cluster into three repeatable patterns. Context degradation: As an agent iterates, its context window fills with tool logs, intermediate observations, and prior outputs. Attention dilutes; the well-documented "lost-in-the-middle" effect: early system-prompt instructions get buried under noisy downstream content, and hallucination probability compounds with every additional turn. Schema drift: Tell an LLM to "always respond in valid JSON matching schema X" and it will comply, for a while. Native structured-output modes from model providers help, but don't eliminate the problem. Under context pressure, by iteration 14 it inserts a trailing comma, wraps a value in a stray Markdown fence, or silently turns an array into a comma-separated string. Your parser doesn't get a warning. It gets an unhandled exception in production. Infinite loops: A tool returns a 500 or an empty payload. An unbounded agent, with no explicit circuit breaker, will often just re-issue the identical call with identical parameters, indefinitely, until it burns through your token budget or hits a context ceiling. None of these are exotic edge cases. They're the default behavior of an agent that owns its own control flow. Decouple Cognition From Control Flow The fix is architectural, not prompt-level: the LLM should act as a stateless processing node, transforming input or picking from an explicit list of valid next states, while a hard-coded, deterministic orchestrator owns the loop. Three patterns do most of the work Strict output enforcement: Never parse raw LLM text with regex or a loose json.loads() in production. Enforce a schema at the inference boundary (Pydantic, Zod, protobufs) and reject anything that doesn't conform before it ever reaches application logic. State-machine transition routing: Instead of asking the model "what should we do next," define an FSM where states are real code functions and edges are explicitly allowed transitions. The model doesn't choose freely; it selects from an enumerated set of legal next states. This also pays off at debugging and audit time: every failure resolves to a specific illegal transition or a tripped breaker, both of which show up cleanly in your logs, instead of an unstructured stochastic trace you have to reconstruct by hand. Idempotent execution and circuit breaking: Treat every tool call like an untrusted third-party RPC. Give every state transition an idempotency key, persisted alongside the state itself, so a retried write can't double-charge a card or duplicate a record, and so a crashed worker can resume a transition instead of replaying a side effect that already committed. If the same transition fails Ntimes with the same bad input, trip the breaker and route to a deterministic fallback, a cached response, a rule-based heuristic, or a human-in-the-loop queue. Here's the shape of it in code: a validated transition handler with structured error feedback and a persisted idempotency key: class DeterministicExecutor: def __init__(self, store: StateStore, max_retries: int = 3): self.store = store # persistent, not in-memory self.max_retries = max_retries self.allowed_transitions = { AgentState.INIT: {AgentState.QUERY_DB, AgentState.ESCALATE}, AgentState.QUERY_DB: {AgentState.FORMAT_RESPONSE, AgentState.ESCALATE}, } def execute_transition(self, run_id: str, prompt: str) -> bool: current_state = self.store.load_state(run_id) idempotency_key = f"{run_id}:{current_state.value}" # If this transition already committed (e.g. worker crash-recovery), # skip re-execution and return the prior result. if self.store.already_committed(idempotency_key): return True for attempt in range(1, self.max_retries + 1): raw = llm_call(prompt) try: proposal = StateTransitionProposal(**json.loads(raw)) if proposal.next_state not in self.allowed_transitions[current_state]: raise ValueError(f"Illegal transition: {proposal.next_state}") validate_payload(proposal.next_state, proposal.payload) self.store.commit(idempotency_key, proposal.next_state, proposal.payload) return True except (json.JSONDecodeError, ValidationError, ValueError) as err: prompt = f"{prompt}\n[SYSTEM ERROR]: {err}. Correct the JSON format." # Circuit breaker: fall back to a safe, human-reviewed path self.store.commit(idempotency_key, AgentState.ESCALATE, {}) return False Every failed attempt feeds a structured error back to the model instead of failing silently. Every success is a legal, pre-validated transition, durably committed, never a free-form guess, and never lost if the worker process dies mid-flight. Four Rules That Hold This Together The orchestrator owns the loop: Never let the LLM run its own `while` loop. The top-level loop belongs to deterministic code, Python, Go, Temporal, Step Functions. The model is invoked *within* an iteration, not in charge of it. Validate at the boundary, sanitize at the node: Raw model output never touches a SQL query, an API call, or a filesystem directly. It passes through a validation model at the exact point it leaves the LLM call. Treat tool calls like foreign RPCs: Assume every call will time out or return garbage. Wrap them with hard timeouts, exponential backoff, and strict interface boundaries, because you should assume nothing about what's on the other end. Design for graceful degradation: Assume the model will fail to conform 1 to 3% of the time, that's not pessimism, it's the observed baseline. Reliability is entirely a function of what happens during that failure spike: a heuristic fallback, a cached result, or a human reviewer, never a silent crash. The Takeaway Prompt engineering alone can't turn a stochastic sampler into enterprise software, no amount of "please always respond in valid JSON" survives contact with iteration 14 of a real agent loop. What does survive is boring, deterministic infrastructure wrapped around the model: schema enforcement at the boundary, an FSM the model can't escape, and circuit breakers that fail safe instead of failing loud. That's not a constraint on what AI agents can do. It's the only way to let them run unattended at a scale anyone would actually trust. References Anthropic, "Building Effective Agents", the foundational distinction between predefined-code-path workflows and model-directed agents. Charles Sieg, "Achieving Determinism with LLM Agents: An Architecture Guide", on building deterministic scaffolding around a non-deterministic model core. "A Methodology for Selecting and Composing Runtime Architecture Patterns for Production LLM Agents", arXiv:2605.20173, which formalizes the stochastic-deterministic boundary as a proposer, verifier, commit, reject contract. Ceaksan, "LLM Agentic Failure Modes", a taxonomy of failure modes specific to agent loops and tool use.