Why Heuristic Detectors Beat LLMs at Finding Agent Failures

TL;DR: We built 20 core rule-based detectors that find failures in AI agent traces. On the TRAIL benchmark (Patronus AI), they achieve 60.1% accuracy vs. 11.9% for the best LLM. Zero false positives. Zero LLM cost. On Who&When (ICML 2025), combined with a single Sonnet call for attribution, they beat GPT-5.4 Mini on both agent identification (60.3% vs. 60.3%) and step localization (24.1% vs. 22.4%). pip install pisama The assumption everyone makes When an AI agent fail...