I Shipped My Code Gate to a Stranger's Machine. Its First Bug Report Fixed the Evaluator Itself.

TLDR: I build a structural gate that refuses AI coding agents’ writes before they land — hard budgets on file size and imports, no human override. Last week it ran for the first time on someone else’s machine, on a real project, with six evaluation questions attached. The report came back with three correct refusals, one false positive, and a fix list. Every fix shipped within a day. The next day, the same machine re-verified every one of them against the code that surfaced them. This is what the loop looked like, including the part where the tool sat dark for two days and nobody noticed. Three weeks ago I wrote about making coding agents work under a constitution: files capped at 500 lines, no more than 7 imports per module, adversarial cross-vendor review, done-means-proven. The system is private while it hardens. But the smallest piece of it — the structural gate that says no before a write lands — travels well. It is one hook, one process, no network, no telemetry. So I packed it as a tarball, wrote six questions into an EVALUATION.md, and put it on a second machine: Windows 10, native, no WSL, a real TypeScript project (a property-market watcher that talks to Telegram), and an agent doing real work on it with a person waiting on the other end. Here is what came back. The report Four days of work. Roughly forty gated write/edit operations. The evaluator answered the six questions and added what I did not ask for: a fix list, ranked by value, with every refusal quoted verbatim. Four refusals. One was a deliberate probe — a 601-line file written solely to check the gate was alive. It was refused before touching disk. Two fired during a real refactor, both on the same rule: [CodeGraph] Fan-out violation: proposed file imports 8 other modules, budget is 7. Consolidate into fewer imports (use a barrel only if it aggregates a coherent concern) or split into smaller modules. Both were correct. One caught a module importing two things whose composition belonged one level down; the other caught a handler re-importing types to describe a context that already had a name. Each fix took about five minutes. The evaluator’s phrasing is better than mine: “Two extra imports do not look like a problem while you are typing them; that is precisely why a machine has to say it.” The fourth refusal was the interesting one — the false positive. The false positive that revealed a missing concept The gate refused to create a file in a session scratchpad — a junk drawer of throwaway analysis scripts — because the directory held 40 files and the god-directory rule says flat piles must become subfolders. The rule is right for a source tree. The target was not a source tree. The gate had no concept of “not a project”, so it graded a junk drawer on architecture, mid-conversation, with a human waiting on Telegram. Cost: ten seconds (the refusal’s own suggestion was followed). But a false positive under live pressure is exactly the kind that gets tools turned off. The fix was conceptual, not a threshold: the rule now checks whether the directory lives under a package.json or a .git before it has opinions about architecture. The part that should embarrass me The gate was dark for the first two days of the evaluation, and everyone involved believed it was running. The install has two steps: a global npm install, and one line registering the hook in Claude Code’s settings. On that machine, both steps hit a permission boundary and were handed to the human. The npm install happened. The settings line did not. No error, no signal — the hand-off was assumed done, and nobody read the file to check. A one-second read would have caught it. Two days of “evaluation” with the brake unplugged. The report’s verdict: “For a tool whose distribution story is ‘install it and it just works’, the first five minutes are the weakest part of the product.” That failure is now a command. codegraph-gate doctor walks all three settings locations, reports which one registers the hook, then fires a synthetic over-budget write through the deny engine and confirms it comes back refused. Exit 0: the brake is on. Exit 1: you are driving without it. One command, and the two dark days become impossible to repeat silently. The fix that operates exactly at the margin The highest-value finding was almost invisible: the gate counted import type toward the 7-import budget. TypeScript erases type-only imports at compile time — no runtime edge, no load order, no cycles — but the counter treated them like any other dependency. One live module reported 7-of-7 where its real fan-out was 5. Why does that matter? Both genuine refusals in the evaluation fired at 8-versus-7. The tool makes its decisions at exactly the margin the inflation was distorting. A budget rule is only as honest as its counter. Fixed: type-only edges no longer count (dynamic import() and require() still do). The evaluator’s borderline module now reports 5. The loop, closed This is the part I actually want to show you. Timeline: Day 1-4: evaluation runs, report lands with four findings and one proposal. Same evening: all four findings fixed, the proposal (doctor) built, new tarball shipped. Next day: the evaluating machine re-verifies every fix against the same codebase and the same scratchpad that produced the findings — not against the changelog. The 7-of-7 module now reads 5. The refused scratchpad (now 60+ files) passes without comment. The audit that double-counted temporary worktrees dropped from 131 scanned files to 74, duplicates gone. The doctor command finds the wiring and proves the deny engine answers. Report, fix, independent re-verification, forty-eight hours. That is the same evidence discipline the whole system is built around — completion claims mean nothing until something that didn’t write the code confirms them — applied to the tool itself, by its first external user. And revision 3 still found a new nit: the install instructions referenced the old tarball name after the version bump. Both real failures of this evaluation were install-step failures. Instruction lines deserve the same care as code. The install line is now version-agnostic and ends with doctor. What the gate did not catch, on purpose During the same period, adversarial review of that project found seventeen confirmed defects the gate passed without comment: a failing command that silently ate the rest of its batch, a watermark that made a bot permanently deaf after a token rotation, a report that double-counted every other night. All seventeen lived inside correctly-sized, correctly-connected files. The gate would not have caught a single one, and it is not supposed to. It measures shape, not correctness. If you buy a structural brake expecting a bug-catcher, you will be disappointed by a tool doing its job perfectly. What it does — and what nothing else in that workflow did — is interrupt at the exact moment momentum is highest: when the file already works, the human is waiting, and one more import is the cheapest thing in the world. Under those exact conditions it said no twice, correctly, and the modules it forced into existence were still paying rent a day later: the next feature and a live debugging session touched only the small files, nowhere near a budget. What I took from it A brake is not a lock. The hook gates the agent’s file tools; a shell redirect never reaches it. The evaluation measured the gap precisely — the gated directory grew from 40 files to 63 through ungated channels, with no dodge intended by anyone. I now say this in the README in exactly those words, because a buyer who reads “blocks” may expect a lock. Install steps fail silently; verification must be a feature. Two of two real failures were install-step failures. If your tool changes behavior of another tool, ship the command that proves the wiring. External evaluation under six blunt questions beats a hundred internal runs. My favorite question turned out to be Q3 — did the agent try to work around it? The answer was no, four times, including under live pressure. That is the number I could never have produced myself. The gate is still private — this evaluation is how it earns its way out. The longer field notes, including the 4 a.m. commit story that started all this, live at mushinnakamoto.substack.com. Want to be evaluator #2? The gate stays private, but the evaluation doesn’t have to. I’m inviting a handful of people to run the exact same thing the Windows machine ran: a private build, five minutes of setup (now ending in doctor), your own codebase, and the same six blunt questions. What I want back is a week of honest answers — especially the false positives and the day you felt like turning it off. If that’s your kind of fun: subscribe at mushinnakamoto.substack.com and reply to the welcome email with the word “evaluate” and what you would run it on. I’ll send the build to the first few that fit. All refusal text quoted verbatim from the evaluation report. The evaluator’s two-day coverage gap is described exactly as it happened — it was the most valuable data point in the whole exercise.