A Weighted Sum Is Not an AND

How one line of arithmetic nearly let any tree in the world open a sealed memory — and what I changed after it did. When I was a kid, my sister and I buried a biscuit tin in the garden. Drawings, a cricket card, a note to whoever dug it up. We drew a map on notebook paper, and then we lost the map. We dug up half the garden and never found it. I've thought about that tin more than I've thought about any photo in my camera roll. Every one of those lives in a scrolling grid, sorted by a timestamp I don't remember, viewable from anywhere, worth almost nothing. A memory that costs nothing to reach costs nothing to ignore. So I spent this Shipaton building the app version of the tin. What I'm building Vantage Point is a native Android app — Kotlin, Jetpack Compose, offline-first, end-to-end encrypted — that lets you bury a memory at a real place. Photos, a voice note, a letter, anchored to a physical object you train the app on. It stays sealed until someone physically stands there and points a camera at it. You can also time-lock a capsule for years, or share one with another person. I started it from an empty directory on 4 August, and it has been a solo build through the whole of Shipaton. I planned it as nine journey-complete milestones and shipped them in order: the vault, capture and burial first; then Open (recognition, proximity, the unboxing ceremony); Pay (RevenueCat — you bury with a "tin", buy more, or hold the Keeper's Key); Share; and Recover (recovery phrase, account deletion, data export). Eight of the nine are done. The ninth is Launch, which is where I am now — release signing, the store listing, and the last of the console gates. The entire product rests on one question: is this the right object, in the right place? This is the story of getting that question badly wrong, and the one change that fixed it. If you are building anything that combines several noisy signals into one confidence score — fraud scoring, liveness checks, sensor fusion, an LLM-as-judge rubric — the same bug is waiting for you. It is very easy to write and almost invisible once written. The naive design is: ask an on-device labeler what's in frame, compare its labels to what was buried, and open if they match. ML Kit's default image labeler, shown a photo of the oak in my garden, returns "Tree". That is true of every tree on Earth. A label match tells you the class of thing you are looking at and nothing whatsoever about which one. As a rough gate it's fine. As an identity check it is worthless, and any product built on it opens on the wrong object with total confidence. So recognition became a composite of three legs: Label agreement — does the live frame reproduce the labels trained at burial? Visual similarity — does it look like the reference images, structurally? Proximity — are you standing where the capsule was buried? Three weak signals. None sufficient alone. That part was right. The bug I combined them the way everybody combines them: a weighted average. Each leg gets a weight, the weights add up to one, the result lands somewhere between zero and one and reads like a confidence score. It ships in a thousand codebases. Visual similarity got the largest share, because it is the leg that actually identifies the object; label and proximity split the rest. Now stand in front of the wrong tree, at the right GPS coordinate. The labeler says "Tree" — a genuine, correct match against the trained label, because the impostor really is a tree. That leg scores full marks. You are standing exactly where the capsule was buried, so proximity scores full marks too. Between them, those two legs carry more than half the weight in the average. So the score is already better than half way to passing before the camera has contributed anything at all. The visual leg — the only one of the three that can tell this oak from the oak beside it — was acting as a bonus on top of an already respectable score, rather than as a requirement. The design document said, in English, the object and the place. The code said the object, or the place, or honestly just a decent showing on any two of them. A weighted average is not an AND. It is a vote — and in a vote, strong legs cover for absent ones. That is the entire point of a vote, and the exact opposite of what a security boundary needs. I want to be clear about how this bug feels from the inside: it does not look like a bug. The weights are principled. The score is in range. The tests pass — because the tests I'd written asserted that a correct match scores high, and it did. Nothing tells you that a wrong match also scores high unless you go and write that test. The fix is a different kind of average Instead of adding the weighted legs together, multiply them. Same three legs, same three weights, completely different behaviour. An average asks how the legs did on balance. A product asks whether every leg showed up. As any one leg approaches zero, the whole result collapses toward zero no matter how well the others scored — there is no amount of label agreement or proximity that can rescue a near-zero visual match, because you are multiplying rather than adding. Run our impostor through it — perfect label, perfect location, hopeless visual match — and the score falls well below the pass bar. And this is the part that matters: it cannot be argued back up. No combination of the other two legs recovers it. The AND stopped being a promise in a design document and became a property of the arithmetic. Containment for free The nicest consequence was one I didn't design. Proximity doesn't fade away gently with distance — it is scored on a curve that is flat at the top for anyone comfortably inside the trigger radius, slides down across the outer half, and hits exactly zero past the boundary. Not "very small". Zero. Multiply a zero into anything and the whole result is zero. Which means: no combination of label match and visual similarity can open a capsule from outside its radius. Not "shouldn't". Can't. Not because a guard checks the distance, but because there is no path to a passing answer. I could have written that as an early return when the distance exceeds the radius. It would work today. It would also be one refactor away from a caller that computes the score without calling the guard — and that caller would introduce a silent, unfalsifiable regression in the one property the product cannot afford to lose. Encoding containment in the shape of the calculation means there is no code path to forget. Then the multiplication bit back Here is the trade, and nobody tells you about it: a score that can be zeroed by one leg will be zeroed by one leg. Including by your own bugs. 1. Absence is not zero Training takes three deliberately different angles — straight on, from the side, from above — and I kept only the labels that appeared in all three. In practice, the overhead shot of a tree shares essentially nothing with the front-on shot. The set of labels common to all three came back empty. An empty trained label set scored the label leg at zero. Multiplied through, that made the whole score zero — for a capsule you were standing directly in front of, with a perfect visual match, forever. An anchor that could never be opened, at any distance, from any angle, for the rest of time. The lesson generalises well beyond this app: "the signal disagrees" and "this signal has no data" are different facts, and conflating them is fatal. A missing leg must be dropped, with its weight redistributed across the others — never scored zero. The rule is now a majority instead of unanimity: a label has to appear in at least half the trained angles. That keeps the original intent — a label seen once describes where the photographer was standing, not the object — without the cliff. 2. Subtract chance before you weight anything Visual similarity is a perceptual hash of the image's structure, plus a colour histogram, weighted heavily toward structure. Colour is corroboration, never identity: two different trees in the same garden have near-identical colour profiles — same bark, same foliage, same sky. But raw agreement is not evidence. Two completely unrelated perceptual hashes still agree on about half their bits, purely by luck. Two unrelated outdoor photographs overlap heavily in colour. A pair of random images therefore looks "50% similar" if you take the raw number at face value — so both signals are now rescaled against that chance level before they earn any weight, and pure coincidence scores zero rather than a half. Treating a coin-flip as "50% similar" is precisely how a recogniser opens on the wrong object while sincerely reporting high confidence. If you take one thing from this article for your own scoring code: find the chance level of your metric and subtract it. Most people never do, and the resulting scores read as far more meaningful than they are. 3. Don't charge the user for your model's modesty ML Kit's labeler is calibrated conservatively. A correct, unambiguous "Tree" comes back well short of certainty and essentially never at full confidence. Feeding that raw number in meant a perfect label match still capped that leg below full marks and dragged the whole score down — eating most of the margin between "hold steadier" and an opened capsule. The user was paying for the labeler's humility. There's now an attainable-certainty ceiling: past the point where this model is as sure as it ever gets, the leg scores full marks. Below it the leg still slides, so a hesitant guess is still worth less than a confident one. 4. A coverage metric punishes you for collecting more data The label leg measures coverage — the share of the trained labels that the live frame reproduces. Which means a long trained list makes the leg unreachable: keep fifteen labels, have a live frame at a different time of day reproduce six, and you score badly on a leg that should be shouting yes, obviously, this is a tree. It's now capped at the few most confident labels. A handful of labels describe the object. The tail describes the weather and the background. Two bugs that had nothing to do with the maths GPS noise nearly shipped. Proximity started as a straight ramp down to zero at the boundary. Under tree cover, a fix accurate to 10–15 metres put the reported distance anywhere from 4 to 22 metres while the user stood perfectly still. That leg swung wildly several times a second — and once the legs are multiplied, that swing is the difference between an opened capsule and "not quite". Hence the flat plateau: comfortably inside the radius, standing at the place scores like standing at the place, full stop. The failure ladder ran faster than a human. The design said three consecutive failures before offering an alternative way in. Recognition completes in tens of milliseconds, so "this place may have changed" arrived before the user had finished raising the phone. Scoring is now rate-limited to a few frames a second, and the ladder advances at most once every couple of seconds. Rate-limiting something because it was too responsive was not on my Shipaton bingo card. The uncomfortable part The pass bar is set above where a same-class impostor at the right coordinate lands. For a long time, my code comment quoted a comfortable gap between the two. That comment went stale. Every improvement I made to the correct-match path — the certainty ceiling, the label cap, the proximity plateau — also helped the impostor, because an impostor is by definition something that scores well on the coarse legs. The gap is now roughly half what the comment claimed. That margin is thinner than it was, and I'd rather write it down than keep a flattering number in a comment. The asymmetry is what makes it tolerable: refusing a genuine match costs a user one step closer and another try. Accepting an impostor opens a sealed memory at the wrong object, in front of the wrong person, and cannot be taken back. When the two errors cost different amounts, the threshold belongs nearer the expensive one — and the margin gets re-measured before any weight moves again. Every number in this article lives in one dependency-free file, unit-tested on the JVM with no Android on the classpath at all. The camera module only supplies pixels. Recognition quality is the thing most likely to need retuning after real-world use, and retuning it should never mean editing an image analyser and hoping nothing else moved. What I'd tell you to steal If your rubric says AND, don't write a weighted average. Multiply. An average is a vote, and votes let strong signals cover for absent ones. Write containment into the arithmetic, not into a guard. A guard is one careless caller away from being bypassed. A zero in a product has no bypass. Absence is not zero. Drop the leg and redistribute its weight. Conflating "no data" with "disagrees" produced the single worst bug in my project. Find the chance level of every similarity metric you use, and subtract it before it earns any weight. Test the impostor, not just the match. My tests all asserted that correct things score high. Not one of them asserted that wrong things score low, which is why the bug survived to production-adjacent code. Pick which wrong answer is cheaper, then say so in a comment. Nearly every hard call in this codebase reduced to that one question. Formulas are policy. The most important safety property in my app isn't a check — it's the shape of an average. Writing it as arithmetic instead of as a guard made it impossible to forget. Somewhere in a garden there's still a biscuit tin nobody can find. Vantage Point is the version where the map doesn't get lost.