Six months ago I shipped a feature that used an LLM to answer customer support questions. It worked great in testing. In production, it told a user that our refund window was 30 days. It's 14. Nobody caught it for three weeks. That wasn't a model problem. That was a me problem. I handed the model a job it was never designed to do — recall specific policy facts reliably — and assumed confidence meant correctness. It doesn't. It never did. Here's what's actually happening under the hood. LLMs d...