Context Windows Don’t Know What’s Still True — I Built a Validity Layer That Does

TL;DRI built a working benchmark for this in pure Python. No APIs, no LLMs, just a deterministic setup with real numbers and a runnable repo.The basic problem is simple. A context window remembers what happened. It does not know whether that information is still valid.So I built two deterministic executors. They do the exact same work. One checks whether a dependency is still valid before acting. The other only finds out after the action fails.That small difference shows up in the numbers. The second executor does work that was already doomed. When your resource budget is tight, that wasted work is enough to fail the whole task.I got one of my main assumptions wrong along the way. I thought graph shape would drive the wasted work. I ran a 96-configuration sweep, and it disproved my guess. Size was the real driver.I updated the experiment instead of forcing my original hypothesis to fit the data. The corrected result was more precise and led to the next experiment.···Same price change, two outcomes. The baseline keeps going and only notices at the moment it acts; the validity-aware executor checks first and replans immediately.This is the diagram I built to show what's actually different between the two executors in this benchmark, without any numbers yet. A flight price is $420, then a plan gets made around that price, then the price jumps to $610. From there the diagram splits into two branches. The baseline branch just keeps executing like nothing happened. It doesn't have any way of checking whether its assumptions are still good, so it walks straight into the stale plan and only discovers the problem when it actually tries to act and the action fails. By then it's already burned a step it didn't need to. The validity-aware branch does one extra thing before acting: it checks whether the price it's relying on is still valid. Since it isn't, it catches that immediately and goes straight to replanning, without ever touching the doomed action. I drew this before writing a single benchmark number, because I wanted the mechanism to be obvious on its own before I started asking anyone to trust a chart. Everything after this image is just putting numbers on the gap between these two paths.A context window can remember everything about the past while giving an agent the wrong picture of the present.This is easy to miss because it does not look like a memory failure. Nothing was forgotten. No text was cut off. No logs showed a missing record. The fact that drove the wrong decision was right there in the window. It just stopped being true three minutes ago, and nothing in the baseline executor was checking for that change.Take a basic case:10:00: Flight A costs $420.10:01: The plan is to book Flight A.10:03: Flight A jumps to $610.10:04: The system still books based on the $420 plan.If you have ever debugged an agent that confidently followed a plan built on a broken assumption, this is one failure mode you may have encountered. It is not context loss. It is context that stays around long after it stops being valid.I spent the last few weeks building a benchmark to measure what this actually costs in wasted steps and failed tasks. I also wanted to see if tracking validity, instead of just keeping facts in memory, fixes the problem.To be clear about the title: "knows when context goes stale" does not mean the system predicts the future. It simply means the system checks if a fact is still valid right before an action relies on it. It re-verifies the assumption instead of moving forward blindly.···Presence Is Not the Same as ValidityMost talk about context windows focuses on space. People worry about having too much context, not enough context, or loading it in the wrong order. Those are real problems. But they are not the problem here. This issue is all about time.A context window is basically a transcript. It records what happened in order. A transcript tells an agent about the past. A validity layer tells the agent if that information is still safe to use right now. A normal context window cannot do that second job. It just was not built for it.To make this concrete, I gave every fact in the benchmark one of four states instead of a simple true or false:ACTIVE: Current evidence supports it.STALE: It was true once, but newer data exists.SUPERSEDED: A newer observation completely replaced it.UNKNOWN: There is not enough evidence to say either way.That fourth state matters a lot. Keeping UNKNOWN separate from an outright failure gives the executor a third choice. It can verify the fact before acting or making a new plan. This lets the benchmark measure the real cost of uncertainty. It stops treating every doubt as a hard failure, which ended up being one of the most interesting results.There is another distinction worth calling out because it drives the more complex failures in this benchmark. We need to separate factual invalidity from operational invalidity.A fact is factually invalid when it simply becomes false. The flight price is a clean case. It was $420, and now it is not. That is easy to spot. It is what most people picture when they think of stale data.A fact is operationally invalid when it is still true but can no longer support a decision. Imagine a database holds exactly 10,000 records. That number has not changed. But the database itself just went offline. The record count did not become false. It just became completely useless. Truth and usability are different things. That distinction matters here because a fact can remain true while the dependency that makes it usable has failed.This is why a validity system cannot just slap a timestamp on every log entry. A timestamp only tells you how old something is. It does not tell you if the system that made it usable is still working. An agent that only checks timestamps will happily act on a fresh record count from a broken database. Nothing about the count's timestamp changed. The actual break happened somewhere else in the dependency graph, and the record count was just sitting downstream.···How This Fits With My Other WorkI have spent a lot of this year building systems to fix different context failures. We need to be clear about what this specific problem is. These are completely separate issues, not just variations of the same theme:Too much context in the window: Fixed with pruning and compression.Retrieving the wrong context: Fixed at the retrieval layer.Context placed in the wrong spot: This causes the "lost in the middle" effect [1]. Context scoped too broadly: Like grabbing a whole codebase instead of just the relevant slice. A static compiler fixes this by narrowing the scope before retrieval even starts.Context decaying over a long session: Fixed with proper memory management.None of those solutions fix a fact that was completely right when it loaded, but silently became wrong later. Nothing flags that change. That is exactly what this article covers. It is a totally new angle, not just a repeat of old work.···Why This Is Not an LLM BenchmarkEvery benchmark I build runs on pure Python with no API calls. This one is no different. But the reason matters a lot more here.If I used a real LLM for this test, every result would need a footnote. Did the task fail because of the state tracking I am actually testing? Or did the model just mess up its reasoning? Was the prompt slightly off? Did the API provider have a bad day?There are simply too many variables. This experiment is strictly about tracking state, not testing model quality.So I completely isolated the mechanism. I built two deterministic executors. They are not open-ended AI agents. They are just small state machines. They simply look at the data and decide whether to continue, verify, or make a new plan:The Baseline Executor: This one just runs the next step in its plan. It keeps moving forward until an action completely fails. It is not blind. It will eventually find every broken dependency. It just finds out way too late, exactly when the action actually breaks against the real world.The Validity-Aware Executor: This one checks the status of a dependency before running a step. If the data is ACTIVE, it executes. If it is SUPERSEDED, it makes a new plan immediately. If the data is STALE or UNKNOWN, it spends one step to verify the fact. Then it acts based on what that verification actually finds. Simplified from the actual executor logic:Both executors get the exact same task. They face the exact same sequence of world changes. They pay the exact same cost to recover once they need a new plan. The only difference is when they realize something broke.I made sure this was not just an assumption. I got hooked into the world-building code directly. I confirmed both executors start with the exact same facts and event schedules in every single scenario.There is one more important design choice. It is an easy one to get backwards. The benchmark keeps the actual ground truth completely separate from what each executor currently believes.Neither executor can read the true state of the world directly. The real world only becomes visible when an executor actually takes an action or pays for a verification step. This split makes the whole benchmark honest. If the validity-aware executor could just peek at reality, it would win every single time. But it cannot. It has to figure things out on its own.···When This Actually MattersTracking validity is not free. Experiment 3 below shows exactly how much it costs. Because of that, we need to be clear about when you should actually build it.You should use this for multi-step plans where actions have a real cost. Think about tool calls, API spend, or side effects you cannot easily undo. It is also needed for long sessions. If minutes or hours pass between learning a fact and using it, data goes stale. It also matters when you have a hard limit on resources like tokens or latency. In those cases, wasted steps do not just slow you down. They cause the whole task to fail.Skip this for single-shot queries. Facts do not have time to go stale there. Skip it if your actions are cheap and easy to retry. If failing costs nothing, then finding out late costs nothing too. You can also skip it for static tasks. If you are just doing reference lookups and facts never change, this whole problem does not apply to you.If your agent makes small, fast, and cheap moves, do not bother. This solves a problem you do not have yet.···What I Actually MeasuredI focused on five key metrics. I ranked them by how much weight they carry:Pre-Failure Work (PFW): This is the headline number. It counts how many steps run after a dependency breaks but before the system catches it. This is doomed work. The system wastes computation on a plan that is already dead.Stale Context Utilization Rate (SCUR): This tracks decisions made using context that is actually stale. I measure this against ground truth, not what the executor believes. It deliberately ignores false alarms. Experiment 3 shows exactly why that distinction matters.Recovery under a fixed budget: This simply measures if a task can still finish within a strict resource limit after recovering from a failure.Verification Count: This counts how many times the validity-aware executor paid to check an uncertain fact.Execution Overhead: This is the control stat. It shows the pure cost of the validity mechanism when nothing ever breaks.I compute the step budget exactly once before injecting any faults. The budget relies entirely on the graph structure and which fact is at risk. It never looks at when a fault actually fires or which executor wins.The formula is very simple:That last term is a deliberate choice. The recovery allowance needs to scale with the worst-case scenario. A flat constant would starve larger graphs of the room they need to recover. Planning around a specific at-risk fact is fair. Knowing the actual outcome of that plan is not. The formula never sees the outcome.···Experiment 1: Does Stale Context Actually Cause Wasted Work?I started with a basic test. It is a four-step chain where a key fact changes value just one step into the run.Here is what happened:MetricBaselineValidity-AwareSteps used96Pre-Failure Work20Replans11SCUR0.750.50CompletedYesYesBoth executors finish the task. Both pay the exact same cost when they need to make a new plan. The only real difference is when they spot the broken dependency.The baseline runs two extra steps on a dead plan before the failure finally hits. The validity-aware executor catches it right away when it checks dependencies and sees the superseded state.Do not call this "33% faster." That misses the point. What actually happened is cleaner: the validity-aware executor did zero doomed work, while the baseline did two steps of it. Within the parameters of this deterministic benchmark, tracking state validity eliminated 100% of pre-failure steps. Wiping out doomed work is just built into how the mechanism works.···Experiment 2: How Big Is the Damage, Structurally?This is where the project took a turn. The correction ended up much more useful than my original hypothesis.I started with a simple dependency chain (D1) with a depth of 1 through 10, injecting a fault right at the start.Baseline PFW: 0, 1, 2, 4, 9Validity-Aware PFW: 0 across the boardThat is clean, but I want to be upfront. On a single-path chain, this behavior follows directly from how the two policies are written. It describes the mechanism rather than revealing a hidden truth about the world.Next, I looked at branching topologies (D2). I wanted to know if graph shape mattered beyond raw depth. I built three topologies sharing a single root fact: a chain, a shallow/wide tree, and a deep/branching tree.TopologyNodes AffectedBaseline PFWChain (depth 6)65Shallow/wide (depth 4)2524Deep/branching (depth 8)3029At first glance, this looked like proof that shape drives the cost. The shallow graph had less depth than the deep one but wasted almost as much work.I wanted that to be the finding. A "shape matters" conclusion makes for a much better story. But before writing anything based on three hand-picked examples, I stress-tested it. I ran a sweep of 96 configurations across different depths, branching factors, and merge factors to see if the rule held up everywhere.It did not. Every single configuration followed the exact same formula: Baseline PFW = total affected nodes minus 1. There were zero exceptions across all 96 runs.For this generator, shape did not affect PFW once the affected closure size was fixed. Size did. In this generator, one shared fact touches every node in the graph no matter how it branches. What looked like a shape effect in three carefully chosen examples was just three different sizes wearing different outfits.I would rather report the correction than keep the more dramatic claim. The numbers speak for themselves:96 configs tested. baseline_pfw != total_nodes - 1 in 0 of them.That is not a disappointing result. It is a more precise one. It also explains why the next experiment had to happen. If size drives the cost, the real engineering question is not how deep your dependencies go, but how much of the plan a single fact actually touches. Can you shrink that blast radius on purpose?So I ran a third test with isolated branches (D3). I asked what happens when a fact does not touch the entire graph. I built several independent branches. Each branch used its own private fact before merging into a shared final segment. If you break one branch's fact, the others should stay untouched.BranchesTotal Plan SizeExposed NodesExposed %Baseline PFW188100.0%7426830.8%7850816.0%7This is the structural result I found most useful. As the total plan grows from 8 nodes to 50, the absolute damage from one broken fact stays flat: 7 doomed steps every time.What shrinks is the fraction of the plan exposed to danger. Breaking a plan into independent branches does not make any single failure cheaper. Instead, it makes the overall system more resilient because unrelated work stays unrelated.If this concept sounds familiar from my earlier work on dependency graphs, it is the same idea. We are just applying a structural blast radius to the execution state instead of prompt components.The plan grows from 8 to 50 nodes across these three runs, but one broken fact still only costs 7 wasted steps every time. What shrinks is how much of the plan that failure actually touches.This chart is the payoff for a question I only started asking after the previous experiment surprised me: if a stale fact doesn't care about shape, does it at least care about isolation? I built three versions of the same plan, splitting it into 1, 4, and then 8 independent branches that all feed into one shared final step. Then I broke the same single fact each time and watched what happened. The blue line is the one number that never moves: the actual damage from that one broken fact, in wasted steps, stays exactly 7 no matter how big the surrounding plan gets. The purple bars are the number that does move: as more of the plan exists outside that one branch, the fraction of the plan actually exposed to the failure drops from 100% all the way down to 16%. Nothing about the failure itself got smaller. What got smaller was how much of the plan had to care about it. That's the difference between a mistake that takes down everything downstream and a mistake that stays where it happened, and it's decided entirely by how the plan is structured, not by anything about the fact that broke.The Formula Underneath All ThreeOnce I had all three results side by side, I wanted to see if they were secretly the same thing. They are.In the cases covered by the verification script, the PFW values follow the same relationship:I wrote a small verification script to test this. It calculates the expected PFW from the graph structure alone, without running the executor at all and compares it against actual outputs.It matched all 15 cases I threw at it:The 5 chain depths from D1.4 timing variants of the value-change scenario, where I injected the fault at steps 0 through 3 to test the timing half of the law.The topologies from D2 and D3.D1 and D2 are the same underlying relationship viewed through two different lenses. D3 just asks the same question for isolated dependencies. Varying when the fault happens instead of changing the shape of the topology is just the formula accounting for a different variable. It held there, too.Every one of 96 different plan shapes, plus a separate chain depth sweep, lands on the same line. Wasted work isn't influenced by shape once you know the size.This is the chart that made me go back and rewrite a section of this piece I'd already drafted. I'd built three example plans, a chain, a wide one, and a deep branching one, and the wide one looked like it was wasting almost as much work as the deep one despite being much shallower. That felt like a shape effect, and I almost wrote it up as one. Instead I generated 96 more plans across a whole range of depths and branching patterns, plus five plain chains at different lengths, and ran every single one through both executors. Every single point landed on the same dashed line. Not close to it, on it. The line is just plan size minus one. It didn't matter if the plan was a long thin chain or a short wide tree or something in between, once you knew how many steps depended on the broken fact, you knew exactly how much would get wasted. Shape had nothing to do with it. The flat blue line along the bottom is the other half of the story. However big the red line climbs, the validity-aware executor's wasted work never leaves zero, because it never gets far enough into a doomed step to waste anything on it.···Experiment 3: Is Checking Always Worth It?If validity-aware execution always won, I would not trust the benchmark. So I built a case where verification could plausibly add cost without preventing a failure.An observation makes a fact ambiguous rather than explicitly wrong—a signal saying, "this may have changed." I ran it two ways: once where the underlying fact genuinely did change (E1), and once where it stayed true as a false alarm (E2).ScenarioBaselineValidity-AwareE1: Real Change7 steps6 steps (plus 1 verification)E2: False Alarm3 steps4 steps (plus 1 verification)On a real change, checking pays for itself. One verification step buys early detection and avoids a doomed action.On a false alarm, that same verification is pure cost. The baseline never noticed the ambiguity and paid nothing. The validity-aware executor spent an extra step confirming something that turned out to be completely fine.This is the main takeaway from this section: checking freshness is never free. The real engineering question is not "should you verify?" Instead, it asks whether the expected cost of acting on stale state outweighs the cost of checking. That answer depends entirely on your domain, not on the mechanism itself.···Experiment 4: Does Any of This Actually Matter?Every scenario above eventually completes for both executors if I give them enough budget. So there is an obvious question: if both systems get to the right answer, does the difference actually matter?I swept the step budget on the value-change scenario from generous to tight and left everything else unchanged:BudgetBaselineValidity-Aware9CompletesCompletes8FailsCompletes7FailsCompletes6FailsCompletes5FailsFails4FailsFailsAt budgets 6 through 8, the validity-aware executor still finishes. The baseline doesn't. It spends two steps on work that became useless, then still needs to recover and finish the task. With less than nine steps available, there isn't enough budget left.I didn't tune the budget to create this gap. I swept it and found where the two executors started behaving differently. If I had to pick one chart from the whole benchmark, this would probably be it.Same task, shrinking budget. At budgets 6 through 8, the validity-aware executor still finishes; the baseline runs out of room before it can recover.Every other chart in this piece shows one executor doing less wasted work than the other. This is the one where that difference actually decides whether the task gets done at all. I took the same value-change scenario from earlier and reran it under a shrinking step budget instead of a fixed one, from generous down to tight. Both lines start at the top: with enough room, both executors finish the task. But the baseline needs its full unconstrained cost to make it, because it has to pay for the wasted work it did before noticing the problem, and then pay again to recover from it. Give it anything less than that, and it runs out of steps before it gets there. The validity-aware executor never did the wasted work in the first place, so it has room to spare at exactly the budgets where the baseline doesn't. I didn't pick 6 through 8 because they looked good. I swept the whole range and that's just where the gap turned out to be. Every other result in this benchmark is really about efficiency, doing the same thing with less waste. This is the one where the same waste turns into a task that doesn't finish.That's where stale context becomes more than an efficiency issue in this benchmark. When there is a hard limit on tool calls, tokens, time, or some other resource, doing a few unnecessary steps can be enough to make an otherwise solvable task fail.···What I'd Still Get WrongA benchmark that only reports wins is not one I would trust either. Here are a few honest limitations:Recovery cost is a flat simplification. Both executors pay the exact same cost to recover, no matter how a real system would handle a new plan. This keeps the focus squarely on detection timing, but it completely ignores the messiness of actual replanning strategies.Verification is binary and devoid of nuance. A check always resolves fully—confirmed or rejected—at a flat one-step cost. Real verification often gives you partial, noisy, or probabilistic answers instead of a clean green light.The branches are simple chains. In the third experiment, the branches are just linear chains internally. That decouples exposure from total plan size, but it does not separate it from depth within a single branch. To fully decouple size, depth, and breadth at once, you would need to combine complex fan-out structures with isolated branches. I have not done that yet.This is deliberately not an LLM benchmark. As I mentioned earlier, I am not claiming that an actual language model behaves exactly like a deterministic state machine. The goal here is narrower: to isolate the cost of acting on state that was valid when observed but became invalid before execution, without introducing model behavior into the measurement.···What This Actually Buys YouPulling every scenario into one table:ScenarioWhat It TestsBaselineValidity-AwareC (Control)Overhead when nothing breaks5 steps5 steps (0 overhead)A (Value Change)Basic early-versus-late detection9 steps, 2 doomed actions6 steps, 0 doomed actionsD1 / D2 (96 configs)How does PFW scale with the invalidated dependency closure?PFW = affected closure size minus 1 in the tested generatorPFW = 0, alwaysD3 (Isolation)Does damage scale with unrelated growth?7 doomed actions, fixed, as plan grows 8 to 50 nodes0 doomed actions throughoutE1 (Real Change)Does checking pay off when right?7 steps6 steps, including 1 verificationE2 (False Alarm)Does checking cost anything when wrong?3 steps4 steps, including 1 verificationBudget SweepDoes it change the outcome, not just the cost?Fails at budgets 8, 7, and 6Completes at budgets 8, 7, and 6No row here claims an unconditional win. E2 is the single place where the validity-aware setup spends more than necessary. That exact trade-off is the reason the other six rows are worth believing.···The TakeawayEvery number in this piece comes back to one simple idea: context has history. Agents need state.A transcript tells you what happened. It is usually an excellent transcript, too. Nothing was lost, nothing was truncated, and every fact stayed right where it was supposed to be. But making a decision requires answering a completely different question than the one a transcript is built to answer. It needs to know what is still true right now, and a context window by itself has no mechanism for telling you that. It was never designed to.That is not a criticism of context windows. They do exactly what they are built to do: hold everything that happened, in order, so nothing gets dropped. The gap is not in what they store. It is in whether the system tracks the validity of what it stores as the world changes.The problem with AI agents is not just that they run out of context. Sometimes the more subtle failure is the exact opposite. They remember something perfectly, long after it stopped being true, and nothing in the transcript tells them to stop trusting it.More context does not fix that. A bigger window just gives a stale fact more company. What actually helps is an older systems idea [2]: invalidate state when it is no longer safe to use, and detect that before acting rather than after failure.···CodeFull benchmark, all scenarios, the 96-configuration parameter sweep, the closed-form verification script, and the figure generator: https://github.com/Emmimal/context-validity-benchmark/Everything under context_validity/ (facts, graph, world, executors, scenarios) runs on the Python standard library only: no API keys, no external packages, and deterministic benchmark inputs and execution. make_figures.py is the one exception (matplotlib), used only to render the charts in this article from already-computed results. Clone the repo and run python run_all.py to reproduce every number in this piece, or python verify_pfw_law.py to reproduce the closed-form law check directly.···References[1] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, "Lost in the Middle: How Language Models Use Long Contexts," Transactions of the Association for Computational Linguistics, vol. 12, pp. 157–173, 2024, doi: 10.1162/tacl_a_00638.[2] Karlton, P. (attributed). “There are only two hard things in computer science: cache invalidation and naming things.” No original written source is known; attribution is confirmed by Karlton’s son, David Karlton, in “Naming Things Is Hard,” karlton.org, December 4, 2017: https://www.karlton.org/2017/12/naming-things-hard/···DisclosureAll code, benchmark results, and figures in this article are original work, developed and tested on Python 3.12, CPU only, no GPU. The benchmark results reported here are reproducible by cloning the linked repository and running run_all.py, except where the article explicitly frames something as a derived or analytically-checked value (the closed-form PFW law) rather than a direct executor run. The benchmark's core (context_validity/) uses the Python standard library only; matplotlib is used solely by make_figures.py to render the article's charts from already-computed results, not by the benchmark logic itself. The featured image was generated with Google Gemini. I have no financial relationship with any tool, library, or company mentioned in this article.···