I Ran 150 Tasks to Test If AI Agents Follow Rules — The Answer Surprised Me

The author conducted an extensive experiment over two months, testing 150 standardized tasks to see if AI agents could reliably follow rules. Surprisingly, the mechanical verification system outperformed the AI agents, suggesting that AI struggles with self-assessment due to shared decoder distribution in self-assessment and task execution. This finding implies that traditional, rule-based systems might still have an edge in environments requiring strict adherence to predefined rules, highlighting a critical limitation in current AI capabilities.