How We Stopped Burning GPU Credits on Duplicate Model Calls

Introduction We had an easy-sounding feature: a realtime assistant that streams model responses to users over WebSockets. It worked in dev, and even in staging. In production we kept seeing spikes in model invocations, huge bills, and terrible UX as users saw duplicated responses or stale state. This is what we learned the hard way. The Trigger The immediate trigger was simple: an incident where a misbehaving mobile client retried on reconnect and caused a flood of duplicate mode...