My Safety Checks Were Monuments to Bugs I Had Already Fixed

I am building Kotohira Digital Residents for Shipaton 2026 — a membership app for a Japanese town of about 8,000 people. Expo SDK 57, React Native 0.86, react-native-purchases, TypeScript. The whole product is one idea: A ¥5,000 annual subscription that cannot buy the thing you subscribed for. Payment opens a resident card. The card only completes when you have also helped somebody in the town three times. RevenueCat grants the entitlement; the three acts are a separate condition that money does not touch. Over about three weeks I found six bugs in this codebase that I would file under completely different labels — a dedupe predicate, a data verifier, a platform branch, a navigation graph, a prompt payload, a key-handling policy. Then I noticed they were the same bug six times. This post is about the shape they share, why my tests could not see it, and the three checks I ended up with instead. (Code below is real, from commit b01ce98. The repository has moved on since — where a fragment has been superseded I say so in the text. The comments in the repository are Japanese; I have translated them inline and marked the translations.) 1. The screen that answered correctly and could not be opened The app has an AI guide that answers only from a bundled, verified inventory. It runs on a small VPS process that holds the API key; the client sends a message and a little context. It worked. For weeks it answered over HTTPS. The store screenshot of it was pixel-perfect. Nobody could reach it. The home screen builds its navigation from an array of rows. At one point, while the guide's API was still being deployed, the guide row was excluded on web. The deployment finished. The exclusion did not. Here is the row today, with the comment I left on it: /* * 🔴 This row used to be removed on web only — a leftover from when the guide * API was not yet deployed. It outlived the deployment, so **the web build * had no way to reach the guide at all.** * The screenshot pipeline jumps straight to /guide with pushState, so the * image was fine while the entrance did not exist. That is why nobody noticed. */ { href: '/guide', ja: '案内役に聞く', en: 'ASK THE GUIDE', sub: '確認済みの在庫から、いまのあなたに合わせて答えます', thumb: SPOT_PHOTOS['sp-06'] as number, }, (comment translated from Japanese) The mechanism matters more than the bug. My screenshot pipeline navigates with history.pushState. That renders any route whether or not a single link points at it. A screenshot proves a screen exists. It says nothing about whether a person can get there. I had eight beautiful proofs of eight screens and no proof of a graph. So I added a check. Does the home screen contain links to all seven destinations? It passed. 2. The next bug was two hops away The scanner is the mechanism the entire product rests on. You stand at a venue, you read the QR the organiser posted, your card gets a seal. Its only entrance was a button inside an open-call card. Open calls are only listed for events an organiser has confirmed in writing as qualifying service work, and I have not finished that paperwork. So the list is empty — correctly, honestly empty, with a sentence explaining why. Empty list. No cards. No button. No route to the scanner. I had declared camera permission in the Play data safety form. I had described scanning in the store listing. I had storyboarded it in the demo video. For a screen no user could open. A Play reviewer would have found a permission with no feature behind it. And my brand-new reachability check passed the whole time, because it was measuring one hop and this bug was two hops away. That is the sentence I want to hand you: A check that encodes the last failure will pass the next one. My check asked my question — is the guide linked? — instead of the question that mattered — can a person reach every screen I built? It was not an invariant. It was a monument to an incident. 3. Same shape, twice in one afternoon, in a different subsystem The guide screen displays the inventory it answers from, as its claim to be grounded: 9 routes / 25 inns / 48 stops / N events. The first version counted those from src/data. The model was reading a hand-maintained JSON that contained only the seven events. The screen was citing stock the AI had never been given. Fix: generate the JSON from src/data, and count from the file that is actually sent. Hours later the number was wrong again, because the server did this before building the prompt (as of b01ce98; it has since also learned to drop events marked cancelled or finished): /** Do not hand the model events whose end date has passed. Guards against * forgetting to update `status` in the static inventory. */ function inventoryForToday() { const date = today(); const events = Array.isArray(INVENTORY['催し']) ? INVENTORY['催し'] : []; return { ...INVENTORY, 催し: events.filter((event) => String(event?.end_date ?? '') >= date), }; } (comment translated from Japanese) The static count now matched the file and still did not match what the model received. On 12 September, demonstrating this to the town, the screen would have said seven while the guide could see five. Somebody would have read the number out loud. So I made the screen compute its own figure with the server's rule, relabelled it upcoming, deployed, and wrote this chapter saying the problem was solved. Here is the code I was about to publish as the fix: function upcomingEventCount(): number { const today = new Date(); const jst = new Date(today.getTime() + (today.getTimezoneOffset() + 540) * 60_000); const ymd = /* …format jst as YYYY-MM-DD… */; const events = (INVENTORY as Record)['催し']; if (!Array.isArray(events)) return 0; return events.filter((e) => String((e as { end_date?: string })?.end_date ?? '') >= ymd).length; } const SOURCES = [ /* … */ { photo: SPOT_PHOTOS['sp-14'], label: `これからの催し${upcomingEventCount()}件` }, ]; Read the last five lines again. SOURCES is a module-level const. upcomingEventCount() runs once, when the JavaScript module is first evaluated, and never again. On a phone the module lives as long as the process does. Open the app on 6 September, use it on the 7th, and the screen says six while the guide sees five. The third occurrence was the fix for the second one. I did not find it. A verification pass I had pointed at my own documents found it, by re-deriving the number instead of re-reading my claim about it. The shape holds all the way down. Occurrence one: two sources of truth. My fix unified the source. Occurrence two: the server filtered at prompt-build time. My fix unified the rule. Occurrence three: the rule was evaluated once and frozen. Each fix addressed the incident I had just seen. None of them addressed the property — the number on screen must equal the number in the prompt, at the moment it is read. Here is what is actually deployed: // src/lib/useToday.ts — re-derives on foreground and at midnight export function useJstToday(): string { return useDateString(jstDate); } // app/guide.tsx function upcomingEventCount(today: string): number { const events = (INVENTORY as Record)['催し']; if (!Array.isArray(events)) return 0; return events.filter((e) => String((e as { end_date?: string })?.end_date ?? '') >= today).length; } function Guide() { const sources = sourcesFor(useJstToday()); // … } The date is no longer computed inside the counter; it is passed in, from a hook that re-derives it when the app comes to the foreground and when the clock crosses midnight. Two files that must agree now take the same argument from the same clock. And the check that guards it does not compare totals. It compares the count on each date where the count changes: const ends = [...new Set(inv['催し'].map((e) => e.end_date))].sort(); // …assert the count at every end date and the day after it assert.equal(count(dayAfter(lastEnd)), 0); The old check asserted inventory.length === EVENTS.length. Seven equals seven. It passed during all three occurrences. 4. The older, quieter version: rules that can only subtract Once you can see the shape, the earlier bugs stop looking like different bugs. The dedupe predicate. Activity records were deduped on event ID alone: // don't count the same event twice if (current.some((x) => x.eventId === a.eventId)) return current; Reasonable. Except the qualifying condition is three acts within a year, and at the time only two events in the dataset were classified as volunteer work. Two events, deduped by event ID, means a maximum of two records, forever. I had shipped a membership that was not hard to complete — it was arithmetically impossible. The fix is one &&: if (records.some((record) => record.eventId === next.eventId && record.date === next.date)) { return { records, added: false }; } A festival happens every year. The same event on a different day is a different act. The data verifier. This app shows real people and real businesses. So I wrote a verifier that runs before every public build. One of its rules was: // Photo permission does not extend to product, price, or availability claims. const experiences = declBody(kotohira, 'EXPERIENCES'); if (experiences.slice(1, -1).trim()) { failures.push('unverified experiences/prices/availability remain in public data'); } Read it carefully. It does not check whether an experience is verified. It fails if the array is non-empty at all. I wrote it because of exactly one row — a cooking class whose site copy and booking platform disagreed about the price — and rather than resolve one row, I emptied the table and let the verifier hold the line. Sixteen experiences gone. Then the same reflex took the lodgings, the spots, the photos. Weeks later the app was a membership card, seven calendar entries and ten grey silhouettes. The name blocklist. And the sharpest one. To avoid publishing a real person's name without permission, I had a hardcoded list: const forbidden = [ ['池', '龍太郎'].join(' '), ['近江', '淳'].join(' '), // … eleven more ]; Look at what the code can do. Permission arrives — and the check still fails. The only way to make the build green is to keep the person out. A permission system with no representation for permission. That one had a consequence I did not find for weeks. Four guides who had consented were displayed on the People screen under the heading "experiences with no guide listed", with a note reading "the guide's publication permission has not been confirmed." Both statements were false. All four were in the ledger with every publication channel and a confirmation date — and their names were printed on the same screen, four centimetres below the note calling them unconfirmed. That is what a subtract-only rule does when it runs long enough. It does not just remove data. It starts making false statements about people, in public, in your product's voice. 5. Why these are one bug Line them up: What I wrote What I meant dedupe on eventId never double-count one scan fail if EXPERIENCES is non-empty never print an unverified price a list of forbidden names never publish a name without permission home links to seven destinations every screen must be reachable count from src/data screen and prompt must agree read the signing key from env the key must not leak Every left-hand cell is a description of the last thing that went wrong. Every right-hand cell is a property. I kept writing the left column and believing I had written the right one. A monument is cheap to write, always passes, and feels like diligence. It has two failure modes, and I hit both: It is narrower than the property, so the next instance walks past it (one hop vs. the whole graph). It is one-directional, so satisfying the condition cannot make it pass (the blocklist, the empty-array rule). A rule that can only subtract will eventually subtract everything — including the truth about someone who said yes. Here is the diagnostic I now run on every check I write: If this rule were satisfied tomorrow, could the code express that? And does the rule name a mechanism, or an incident? If the answer is no and an incident, it is not a check. It is a deletion with good manners. 6. What actually replaced them Three techniques. All cheap. All ported to any stack. Crawl the closure; do not hold a list Do not enumerate expected destinations. Walk the built app from its entry point, compute what is reachable, and compare against the routes that exist on disk. Add a screen, forget to link it, build stops. function routesOnDisk() { return readdirSync(join(here, '..', 'app')) .filter((f) => f.endsWith('.tsx') && !f.startsWith('_') && !f.startsWith('+')) .map((f) => (f === 'index.tsx' ? '/' : `/${f.replace(/\.tsx$/, '')}`)); } Two traps worth stealing: Single-page hosts return 200 for paths that do not exist. "Did it load?" cannot tell you whether a screen is real, so a mistyped href passes silently. Treat any link target absent from disk as a broken link, not as a reachable page. I proved both directions by breaking them on purpose: retargeting a row to /scam stops on the dangling link; deleting the links to /scan stops on the unreachable screen and prints the whole graph. React Native Web renders Pressable as a div with role="link". Counting and returns zero. My first dead-end detector reported every screen as a dead end because of this. And distinguish openable from usable. A screen with no way back into the app is a dead end if someone lands on it directly — and an external link is not a way back, it is an exit. My lodging screen has 41 outbound links and not one of them returns. Vary the state and the clock The reason both hop-bugs were invisible is that the guide row vanished by environment and the scanner link vanished by state. A single-configuration check cannot see either, by construction. const STATES = [ { name: 'not registered', paid: false, activities: [], review: false }, { name: 'just registered', paid: true, activities: [], review: false, ... }, { name: 'partway (1/3)', paid: true, activities: SEALED.slice(0, 1), ... }, { name: 'complete (3/3)', paid: true, activities: SEALED, ... }, { name: 'review mode', paid: false, activities: [], review: true }, ]; /* 🔴 Do not run with "today". A check that passes or fails depending on the day you ran it is not a check. */ const DATES = [ { name: 'launch day', date: '2026-09-08' }, { name: 'judging window', date: '2026-10-01' }, { name: 'the day the content runs out', date: '2027-01-01' }, ]; (comments translated from Japanese) Five states × three dates = fifteen crawls per run, currently ten screens each. Two details earn their keep. 2027-01-01 is the day every bundled event has ended — a state nobody has ever seen and that arrives on its own whether or not I plan for it. And the expected set is state-dependent: a paid user must not be able to reach the paywall, so demanding "all screens reachable" uniformly would fail a correct implementation. The check asserts that too. Generate the numbers in your prose from the code This is the one I did not see coming. My verifier only ever read the code. The submission documents — store listing, judge instructions, the demo script for a meeting with the town — were outside every check. So they drifted, and everything stayed green. const CLAIMS = [ { label: 'bundled photos', text: `同梱${SITE_CONTENT.bundled}点`, files: [...] }, { label: 'listing count', text: `${listings}件`, files: [...] }, { label: 'model courses', text: `モデルコース${COURSES.length}本`, files: [...] }, { label: 'store images', text: `ストア画像${shots.length}枚`, files: [...] }, ]; Count the thing in code, then assert that the sentence containing that number exists in the document. It caught a photo count off by one, three screenshot filenames renumbered after the shot list changed, and the inventory figures the guide screen was overstating. If a number appears in your marketing copy, something should be counting it. One more, on the same principle: a rule written in a comment is a monument in prose. My design tokens carried a measured contrast table with the note "placing sub directly on the page background gives 4.35 — not enough for body text." I wrote that, then did exactly that in seven places across three screens. There is no version of me who remembers that note while writing the fourth screen. So I darkened the colour until it passes anywhere (#6B7280 → #5B6472, 4.35 → 5.39) and made the screenshot pipeline measure every text node before it saves a frame. It found the seven immediately, and has since caught a filled button whose white label sat at 18px — 0.66px under the 18.66px threshold where WCAG lets 3:1 count as sufficient. I had written that threshold into the file myself, four hours earlier. And: give the check a way to say yes The blocklist became a ledger the verifier reads: export type ConsentRecord = { id: string; name: string; // must match the screen character-for-character scopes: ConsentScope[]; // app / store / video / site / press / submission / social evidence: 'operator_attestation' | 'release_on_file'; confirmed_on: string; confirmed_by: string; photographer_cleared: boolean; // the subject's likeness and the photographer's // copyright are different rights }; Not on the ledger → build fails. Added to the ledger → build passes. Same strictness, both directions available. Note also what it does not do: it refuses to guess. An earlier version tried to decide whether a string was a trade name or a personal name so it could let trade names through. It now checks one thing — is this in the ledger — because a check that guesses is a check that will guess wrong quietly. The experience rule was rewritten the same way. Instead of "any data fails," it asks whether the data over-claims: // A recorded price conflict must not appear next to a price. if (hasPrice && /一致していません|確認中|表記が二通り/.test(note)) { failures.push(`experience ${id} shows a price despite a recorded conflict`); } // Availability captured at scrape time must not be frozen into copy. // "Not currently bookable" is true the moment you write it and false six months later. if (/予約枠が出ていません|予約できます|満席|受付中/.test(note)) { failures.push(`experience ${id} asserts availability in its note`); } The cooking class now displays without a number and says check the official site for pricing — the true statement I should have written on day one instead of deleting sixteen rows. The catalogue came back: 16 experiences, 25 inns, 9 routes, 48 stops, 115 listings, 108 photos. It had all been reachable the whole time. 7. The one I nearly shipped as a monument to a future incident Last one, because it is the cheapest lesson here. The script that prints the venue poster — QR, layout, A4 sheet — was scheduled to run for the first time on 1 September, days before the event it was for. A tool whose first run is the day it matters is not a tool. It is a plan. I ran it early. It read the Ed25519 signing key from an environment variable and nothing else — which meant that on 1 September, in a hurry, the operator would have been asked to put a private key into a shell environment. Meanwhile a sibling script in the same repository says this in its own header: The passphrase is asked interactively. It is never taken from arguments or environment variables, because those persist in shell history and process listings. Two scripts, one repository, opposite policies — and the stricter one was guarding the lighter secret. Consistency is not something you can hold in your head across a repository; it is something a test holds for you. The generator now reads the git-ignored key file directly, and the test suite generates a poster QR, parses it back, verifies the signature, and confirms that tampering with the date, the event ID or the signature is all rejected. Every run. Not once in September. Because a signature's value is entirely in what it refuses, and I had never once watched it refuse. 8. Where the app actually is It would be against the whole point of this app to overstate it: RevenueCat is integrated for Google Play and a separately packaged Galaxy build. Update, 25 August: the first device purchase succeeded — purchase, restore, cancellation and entitlement expiry all observed on a real handset, using a license-tester account. A sandbox purchase is a test result, not revenue: no production purchase has been made. Update, 27 August: the subscription is no longer on sale. SALES_ENABLED = false in src/lib/purchases.ts; purchase() returns before it reaches the store. Restore is untouched, so a judge redeeming a promo code still gets in. Zero organiser-confirmed activities means nobody can finish the card, and I would rather ship the mechanism with the till shut than charge for a document nobody can complete. Update, 31 August: until today the only evidence that the till was shut in the shipping build was a string search inside the bundle. I installed that build on a handset and looked: the home screen offers no purchase, the paywall says it cannot sell, and the settings screen explains the promo-code route instead. Checking the artefact is not the same as checking the app, and I had been one step short of that for four days. Update, 2 September: the subscription is on sale again. I had asked the organisers whether a build with no purchase flow still qualifies, and on 1 September they answered that the submitted build has to carry a working purchase, though no judge needs to complete one. So SALES_ENABLED is back to true, and the listing now carries the sentence I had been leaving out: while zero calls for volunteers are posted, the annual fee will not finish your card. I closed the till on principle and re-opened it for eligibility, and that is worth writing down rather than leaving for someone else to find. Activity QRs carry an Ed25519 signature and are checked for event, date range, local day, eligibility and duplication. The signature proves the payload came from our key. It does not prove presence, participation, or that an organiser displayed it. There is no GPS. Zero qualifying volunteer activities are currently confirmed. Until an organiser confirms one, no ordinary user can reach 3/3 — and the app says so on screen rather than hiding it. That last bullet is the same failure mode as the dedupe bug: a condition nobody can satisfy. Except this time it is not in the code. It is a phone call I have not finished making, and no crawler is going to catch it for me.