Your LLM Got the Variant Right. But Did It Get It Right for the Right Reason?

I built a benchmark to find out whether a frontier language model can be trusted to interpret clinical genetic variants. The result surprised me, and the way it surprised me is the whole point of the post. The model I tested (Claude Opus 4.8) scored 60 percent accuracy against expert consensus. If I had stopped there, I would have written "the model is mediocre, do not deploy." That conclusion would have been wrong. The real finding only appeared once I stopped measuring accuracy and started me...