Designing Reasoning Boundaries in Agentic Systems

Designing Reasoning Boundaries in Agentic Systems In my previous article - https://hackernoon.com/your-ai-agent-should-not-be-talking-to-all-mcp-tools, I talked about exposing repeatable sequences as domain tools. The main LLM should not have to choose among APIs, queries, and Model Context Protocol (MCP) capabilities. This is a continuation of the article where I will explain internal orchestration and go inside the tool.A high-level tool may call APIs, query databases, apply rules, and invoke another LLM. Which steps belong in software, and which require model reasoning?Hiding five low-level calls behind one function does not help if it makes five model calls instead. The decisions are harder to see, but the latency, cost, and inconsistent behavior remain.Rule of thumb: use software where the result can be computed, and an LLM where it must be interpreted. The hard part is deciding where to draw that line.Three Execution Layers Three layers need to be implemented - Ordinary software handles calculations, validation, permissions, policy rules, workflow state, retries, and timeouts. APIs fail, but calling them does not require model reasoning. Their contracts belong in code. An inner LLM gets a bounded semantic task, such as extracting symptoms from logs or summarizing support history. Its output stays structured and tied to evidence.The main LLM works out what the user wants and explains the result. It should receive compact domain evidence, not implementation detail. User | Main LLM | High-level domain tool |-- APIs and databases |-- calculations and rules |-- validation and retries `-- optional inner LLM | Structured evidence | Main LLM synthesizes and explains Each operation goes to the mechanism with the clearest contract. A service incident example. Suppose an operations agent receives this request: Checkout errors increased after the latest deployment. Find the likely cause and tell me whether we should roll back. The main model could call separate tools for metrics, logs, traces, deployments, flags, and runbooks. A cleaner interface is:diagnose_service_incident(service_id, start_time) Answering that one question means pulling data from six different systems. The mistake is routing every one of those steps through an LLM just because the question sounded like it needed judgment. Look at each step on its own, and most of them are plain lookups or math functions.Retrieve error-rate series -> metrics API Find deployments in the time window -> deployment API Calculate baseline deviation -> code Compare affected endpoints -> code Resolve active feature flags -> configuration API Extract symptoms from unusual logs -> inner LLM Apply rollback policy -> rules Explain evidence and options -> main LLM Only two steps need a model. Fetching metrics, deployments, and flags are lookups. Comparing error rates against a baseline is arithmetic. None of that requires interpretation, so it belongs in code.Logs are different because different systems describe the same failure in incompatible ways. One library reports a connection reset. Another buries the same fault in a stack trace. A third calls it a failed payment-provider request. There's no fixed format to parse here, so this is the one place an LLM should be used - reading messy text and matching it to a category.The rollback decision looks like judgment, too, but it isn't. It's a fixed rule: roll back if errors are more than triple and no migration is running. That threshold should be in code, not in a prompt. The LLM's job is to explain the decision in plain language, not to invent the number.Define the boundary as a contract. A domain tool needs a typed result. Without one, the abstraction just moves a paragraph of text from one model to the next. Practically, this means picking a fixed set of fields that the tool always returns, instead of a block of prose that the main LLM has to re-read and re-interpret. A number stays a number, and a guess stays labeled as a guess, so policy code never mistakes one for the other. The two definitions below show what that looks like for the incident tool. from dataclasses import dataclass from typing import Literal @dataclass(frozen=True)class LogFinding: category: Literal["dependency", "database", "timeout", "unknown"] summary: str event_ids: list[str] @dataclass(frozen=True)class IncidentAssessment: service_id: str error_rate_ratio: float related_deployment_id: str | None affected_endpoints: list[str] log_findings: list[LogFinding] rollback_allowed: bool recommended_action: Literal["rollback", "hold", "escalate"] evidence_ids: list[str] incomplete_sources: list[str] The type separates measured facts from model findings and exposes missing evidence. A trace timeout must not quietly become "no anomalies found."error_rate_ratio is measured; log_findings contains interpretations. Separate fields stop a summary from becoming a metric and let policy code ignore probabilistic evidence. The orchestration can then keep model reasoning narrow: def diagnose_service_incident( service_id: str, start_time: datetime ) -> IncidentAssessment: metrics = metrics_client.get_service_metrics(service_id, start_time) deployments = deployment_client.list_recent(service_id, start_time) logs = log_client.get_anomalous_events(service_id, start_time) error_rate_ratio = calculate_error_rate_ratio(metrics)related_deployment = correlate_deployment(deployments, metrics) affected_endpoints = rank_affected_endpoints(metrics) raw_findings = extract_log_findings(logs)log_findings = validate_log_findings(raw_findings, logs) recommendation = apply_rollback_policy(error_rate_ratio=error_rate_ratio, deployment=related_deployment, findings=log_findings, ) return IncidentAssessment(service_id=service_id, error_rate_ratio=error_rate_ratio, related_deployment_id=get_id(related_deployment), affected_endpoints=affected_endpoints, log_findings=log_findings, rollback_allowed=recommendation.rollback_allowed, recommended_action=recommendation.action, evidence_ids=collect_evidence_ids(metrics, logs, deployments), incomplete_sources=collect_failed_sources(), ) The model is used to do this one job: turn unusual log events into categorized findings with references. It does not calculate ratios, correlate timestamps, or enforce policy. Treat model output as untrusted input. A valid JSON object can still be nonsense. Check model output before the system relies on it. For the log findings above, validation should confirm that: every category is from the allowed set; every event_id exists in the supplied log batch; the summary stays within its length limit; required fields are present; the model did not return instructions or prose outside the schema. I would rather have an event ID than a confidence score. Models can be confident about unsupported claims. Event IDs take reviewers back to the source.If validation fails, retry once, return unknown, or escalate according to risk. Do not pass malformed output to the main model.Design failure behavior before the happy path. A high-level tool pulls several failures into one place. Metrics can time out, logs get truncated, data can go stale, and models return unusable output. Do not flatten them into an empty result. incomplete_sources=["traces"] differs from "the query found nothing." With partial evidence, escalation may be the only honest answer.Set timeouts and retries per dependency, and run independent retrieval calls at the same time. Don't automatically retry non-idempotent operations — their effects may change when repeated.Keep diagnosis separate from execution. A rollback recommendation should not execute unless the user authorizes it and the system checks that authorization.Make every inner LLM call observable. Inner model calls often get lost from the main trace. Then a tool gets slower or more expensive, and nobody can see why. Record at least: model and prompt version; input and output token counts; latency and retry count; schema-validation result; evidence IDs returned; fallback path taken; final tool outcome. Measure total latency, cost, invalid-output rate, escalation rate, and agreement with reviewed incidents. Without those numbers, you cannot show whether the model improves the workflow.Test each layer differently Each boundary can be tested on its own. Rules get unit tests. API adapters get contract and failure-case tests. The inner LLM gets a versioned evaluation set, schema checks, and tests that verify its claims against the source evidence. These tests answer different questions. Keep their results separate, or a good aggregate score can hide a broken component.Testing the pieces separately isn't enough. The main model can choose the wrong tool, and a fluent answer can hide that failure. The complete chain needs its own evaluation strategy.Two ways to draw the boundary badly One bad boundary turns the LLM into a calculator. Date comparisons, thresholds, permissions, and state transitions become slower and harder to reproduce as prompts. A prompt cannot make the same guarantee as a pure function. These costs don't show up right away. Ask a model to compute error_rate_ratio, and it can return a different number on the same inputs from one run to the next, so a decision that should be reproducible turns into a coin flip. Testing stops being a single assertion you trust forever and becomes a statistical evaluation you re-run and hope holds up. Worse, nobody can recompute the number by hand to check the tool's work, defeating the point of having a contract at all.The other forces semantic work into a growing if/else tree. String matching becomes expensive and brittle as variations in logs or intent accumulate. Sometimes a bounded LLM call is simpler.Nested LLM calls are not inherently wrong. Each needs a named job, a consumer, a schema, and a failure policy. If I cannot say what it contributes, I remove it.A practical decision test. Work Default mechanism Example Retrieval API or database adapter Fetch deployment history Exact transformation Code Calculate error-rate change Policy enforcement Rules or workflow engine Permit a rollback Semantic extraction Bounded inner LLM Categorize unusual logs Open-ended synthesis Main LLM Explain likely causes High-impact exception Human approval Authorize a risky action These are defaults. What matters is what the next component needs to trust.Before adding an LLM call inside a tool, ask: Can the operation be specified completely? If an engineer can write exact rules for every valid input, start with code. Does the task depend on semantic interpretation? Unstructured input may justify a model, but the input format alone does not decide. Can the output be validated? Define the schema, allowed values, evidence requirements, and fallback before writing the prompt. What does a wrong answer cost? High-impact decisions may need policy enforcement or human approval even when a model gathers the evidence. What happens when a dependency fails? Make partial evidence visible in the result. Can you measure the benefit? Compare quality, latency, cost, and operational complexity with a non-LLM baseline. Use code where the result can be computed. Use an LLM where the result must be interpreted. Pass typed data between them, validate model output, and record what happened at the boundary.Part 1 was about tool visibility. This article is about where reasoning happens. A production agent needs both boundaries.Even good boundaries don't prove that the system produced the right result.The main LLM might select the wrong domain tool, and that tool retrieves the wrong evidence. A rule might use the wrong threshold, or an inner LLM might return a valid but misleading summary. The final model can turn any of these failures into an answer that sounds perfectly convincing.Evaluating only that final answer won't tell you where the system failed.Part 3 takes up the problem: How do you evaluate an agent that calls tools that call other tools?