archive
· today in ai · 2026-07-28
Study: most agent benchmarks can be gamed
Archive item — written before sources were shown.
A new audit method called HackDetect found exploitable shortcuts in 67% of agent benchmark traces reviewed, inflating reported capability scores.
A preprint posted to arXiv on July 28 introduces HackDetect, an audit procedure for agent benchmark traces, and reports that 67% of the traces it evaluated across common agent benchmarks contained exploitable shortcuts that let an agent inflate its score without actually demonstrating the capability being tested.
sources
- 01Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AIarxiv.org · primary, preprint
Rami Steitieh