TRAIL: Trace Reasoning and Agentic Issue Localization

TL;DR

This paper introduces TRAIL, a comprehensive dataset and taxonomy for evaluating and debugging complex agentic workflow traces generated by AI systems, highlighting current limitations of large language models in trace analysis.

Contribution

It provides a formal taxonomy of errors, a large annotated dataset of 148 traces, and demonstrates the inadequacy of current models for trace debugging in agentic workflows.

Findings

LLMs perform poorly at trace debugging, with Gemini-2.5-pro scoring only 11%.

The TRAIL dataset enables systematic evaluation of agentic trace analysis.

The taxonomy aids in understanding and categorizing errors in agentic systems.

Abstract

The increasing adoption of agentic workflows across diverse domains brings a critical need to scalably and systematically evaluate the complex traces these systems generate. Current evaluation methods depend on manual, domain-specific human analysis of lengthy workflow traces - an approach that does not scale with the growing complexity and volume of agentic outputs. Error analysis in these settings is further complicated by the interplay of external tool outputs and language model reasoning, making it more challenging than traditional software debugging. In this work, we (1) articulate the need for robust and dynamic evaluation methods for agentic workflow traces, (2) introduce a formal taxonomy of error types encountered in agentic systems, and (3) present a set of 148 large human-annotated traces (TRAIL) constructed using this taxonomy and grounded in established agentic benchmarks. To ensure ecological validity, we curate traces from both single and multi-agent systems, focusing on real-world applications such as software engineering and open-world information retrieval. Our evaluations reveal that modern long context LLMs perform poorly at trace debugging, with the best Gemini-2.5-pro model scoring a mere 11% on TRAIL. Our dataset and code are made publicly available to support and accelerate future research in scalable evaluation for agentic workflows.