Decision Logging: The Observability Pattern That Actually Helps AI Agents

Most agent-debugging sessions begin with the same question: Why did the agent do that? Why did it send a notification? Why did it suppress one? Why did it retry the same tool? Why did it act on stale data? Why did a high-confidence prediction turn out to be wrong? Traditional application logs often show that a function ran, an API returned, or an error occurred. They may show that searchHotels completed, comparePrices ran, and sendNotification succeeded. What they often miss is the observable boundary where the system chose one path over another. That is the problem decision logging can solve. Decision logging is not a request to store every prompt, completion, or hidden model rationale. It is a structured record of consequential choices: which policy rule fired, which tool was proposed, why the runtime retried or refused, which source supplied a score, and what outcome was observed later. Events Are Not Explanations Imagine a proactive hotel-price agent. A user reports that it sent an alert for a price drop that did not exist. The logs show: agent session started searchHotels completed comparePrices completed price drop detected generateNotification completed sendNotification completed agent session completed Nothing appears broken. After a long investigation, you discover that a retry read cached inventory outside its freshness window. The comparison succeeded technically, but it compared the wrong observations. The events existed. Their relationship and decision inputs did not. A useful decision record would show that the second path used cached data, identify the cache age and freshness policy, and record the policy result that allowed the notification. That turns “the model made a bad choice” into a testable system failure. A decision log should be structured, queryable, and small enough to emit at meaningful boundaries. It should explain the decision using facts the application can verify. type ConfidenceSignal = { value: number; source: "model_self_report" | "classifier" | "calibrated_model"; calibrationVersion?: string; }; type AgentDecision = { decisionId: string; traceId: string; spanId: string; parentSpanId?: string; runId: string; agent: string; kind: | "intent_classified" | "tool_proposed" | "tool_result_evaluated" | "policy_evaluated" | "action_blocked" | "notification_suppressed" | "response_completed"; occurredAt: string; reasonCode: string; reasonSummary?: string; inputFacts: Record; proposedTools: string[]; executedTools: string[]; confidence?: ConfidenceSignal; costUsd?: number; latencyMs?: number; status: "allowed" | "blocked" | "suppressed" | "failed"; }; The important distinction is between an observable reason and chain-of-thought. A reason code such as PRICE_DATA_STALE or APPROVAL_REQUIRED can be generated by runtime policy and verified from inputs. A short, sanitized summary can help a human understand the event. Do not ask a model to dump its private reasoning into logs and treat that text as ground truth. Free-form rationale may expose sensitive content, be unfaithful to the behavior, and create a much larger security surface. Log the evidence and policy rule the application actually used. Confidence needs similar discipline. A model's self-reported 0.91 is not automatically a calibrated 91% probability. Record the source, and only describe a score as calibrated when you have measured calibration against representative outcomes. Log the Boundary, Not the Entire Runtime This is tempting during development: logger.info("agent decision", { user, prompt, toolResults, modelResponse, }); In production, it can copy personal data, secrets, proprietary documents, and long model content into an observability system. It also makes the useful signal harder to query. A smaller helper can require structured facts: import { randomUUID } from "node:crypto"; type NewDecision = Omit; async function logDecision(decision: NewDecision): Promise { await decisionStore.insert({ ...decision, decisionId: randomUUID(), occurredAt: new Date().toISOString(), }); } Then log a verifiable boundary: await logDecision({ traceId, spanId, runId, agent: "price-watch-agent", kind: "policy_evaluated", reasonCode: "FRESHNESS_WINDOW_PASSED", reasonSummary: "Latest inventory was older than the 10-minute limit.", inputFacts: { inventoryAgeSeconds: 912, maximumAgeSeconds: 600, }, proposedTools: ["send_notification"], executedTools: [], status: "blocked", }); This record explains the behavior without storing the hotel query, user profile, or generated notification. Correlate Decisions With the Trace Decision records become more useful when they participate in the execution trace. A shared trace ID and explicit span relationships let you reconstruct the story: price-watch-agent trace_123 ├─ classify-intent ├─ search-hotels ├─ evaluate-price-change ├─ evaluate-freshness-policy ├─ block-notification └─ respond-to-user OpenTelemetry context propagation (https://opentelemetry.io/docs/concepts/context-propagation/) correlates signals across services by carrying trace and span context. Use those established identifiers rather than inventing a second, disconnected parent-child system. Agent-specific decision events can be attached to the relevant spans or linked to them in storage. Outcomes Happen Later Product outcomes do not necessarily exist when the decision is logged. clicked can be observed when a user clicks. ignored cannot be known immediately; it is an inference made after a defined observation window. Keep runtime decisions and later outcomes separate: type DecisionOutcome = { decisionId: string; observedAt: string; observationWindowHours: number; outcome: "clicked" | "dismissed" | "expired_without_interaction"; }; This avoids rewriting history and makes the measurement definition explicit. It also lets you distinguish delivery failures from relevance problems. Pseudonymous Is Not Anonymous Decision logging can become a privacy risk if identifiers are handled casually. Do not record raw prompts, full completions, email addresses, access tokens, payment details, or private chat history by default. If you need a stable pseudonymous identifier, use a keyed HMAC rather than a plain unsalted hash or a public salt: import { createHmac } from "node:crypto"; const pseudonymKey = process.env.LOG_PSEUDONYM_KEY; if (!pseudonymKey) { throw new Error("LOG_PSEUDONYM_KEY is required"); } function pseudonymizeUserId(userId: string): string { return createHmac("sha256", pseudonymKey) .update(userId, "utf8") .digest("hex"); } An HMAC makes offline guessing harder when the key is protected, but the result is still pseudonymous data. Access controls, key rotation, purpose limitation, and retention rules still apply. OWASP's Logging Cheat Sheet (https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html) recommends excluding or sanitizing sensitive data, credentials, tokens, and session identifiers rather than assuming everything belongs in a log. Store Enough to Debug, Not Everything Forever Decision data grows quickly. Recent records may need detailed incident-level access. Aggregated metrics may live longer. Raw or pseudonymous records should have a documented retention period, restricted access, and audited exports. The right policy depends on the product and legal context, but the engineering principle is stable: collect the minimum evidence needed for a defined debugging or measurement purpose. Decision logging is valuable because it is selective. It records the boundaries that changed the workflow, not every internal token. Questions the Data Can Finally Answer With structured decision events, you can ask: Which policy rules blocked actions most often? Which dependency caused repeated retries? Which tools were proposed but never authorized? Which score sources correlate with later success? Which notifications expired without interaction after the defined window? Which agent or release introduced the first behavioral divergence? These questions connect system behavior to reliability and product outcomes. They are more actionable than a count of log lines or a dashboard of uptime alone. Final Thoughts Agent failures rarely live inside one function. A bad outcome may involve classification, tool selection, stale state, retries, policy evaluation, and a later user response. Decision logging gives the system a structured memory of those boundaries. It should record what the runtime observed, which rule it applied, what it proposed, what it executed, and what outcome was measured later. It should not become a chain-of-thought archive or a shadow copy of user data. When the evidence is structured, correlated, and governed, the question “Why did the agent do that?” becomes much easier to answer.