Holistic Evaluation and Failure Diagnosis of AI Agents

TL;DR

This paper introduces a comprehensive evaluation framework for AI agents that combines top-down and bottom-up analysis, providing detailed failure diagnosis and achieving state-of-the-art results on multiple benchmarks.

Contribution

It presents a scalable, span-level evaluation method that improves failure localization and diagnosis, surpassing prior approaches in accuracy and granularity.

Findings

Achieves up to 38% improvement in category F1 score.

Up to 3.5x better localization accuracy.

Up to 12.5x improvement in joint localization-categorization accuracy.

Abstract

AI agents execute complex multi-step processes, but current evaluation falls short: outcome metrics report success or failure without explaining why, and process-level approaches struggle to connect failure types to their precise locations within long, structured traces. We present a holistic agent evaluation framework that pairs top-down agent-level diagnosis with bottom-up span-level evaluation, decomposing analysis into independent per-span assessments. This decomposition scales to traces of arbitrary length and produces span-level rationales for each verdict. On the TRAIL benchmark, our framework achieves state-of-the-art results across all metrics on both GAIA and SWE-Bench, with relative gains over the strongest prior baselines of up to 38% on category F1, up to 3.5x on localization accuracy, and up to 12.5x on joint localization-categorization accuracy. Per-category analysis shows our framework leading in more error categories than any other evaluator. Notably, the same frontier model achieves several times higher localization accuracy when used inside our framework than as a monolithic judge over the full trace, showing that evaluation methodology, not model capability, is the bottleneck.