The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break

TL;DR

This paper introduces HORIZON, a benchmark for diagnosing long-horizon failures in LLM-based agents, analyzing degradation patterns across domains and proposing a scalable failure attribution method.

Contribution

It presents a cross-domain diagnostic benchmark, a large dataset of agent trajectories, and a trajectory-grounded LLM-as-a-Judge pipeline for failure analysis.

Findings

Identified horizon-dependent degradation patterns in SOTA agents.

Validated a human-annotated failure attribution method with strong agreement.

Provided practical guidance for building more reliable long-horizon agents.

Abstract

Large language model (LLM) agents perform strongly on short- and mid-horizon tasks, but often break down on long-horizon tasks that require extended, interdependent action sequences. Despite rapid progress in agentic systems, these long-horizon failures remain poorly characterized, hindering principled diagnosis and comparison across domains. To address this gap, we introduce HORIZON, an initial cross-domain diagnostic benchmark for systematically constructing tasks and analyzing long-horizon failure behaviors in LLM-based agents. Using HORIZON, we evaluate state-of-the-art (SOTA) agents from multiple model families (GPT-5 variants and Claude models), collecting 3100+ trajectories across four representative agentic domains to study horizon-dependent degradation patterns. We further propose a trajectory-grounded LLM-as-a-Judge pipeline for scalable and reproducible failure attribution, and validate it with human annotation on trajectories, achieving strong agreement (inter-annotator \kappa=0.61; human-judge \kappa=0.84). Our findings offer an initial methodological step toward systematic, cross-domain analysis of long-horizon agent failures and offer practical guidance for building more reliable long-horizon agents. We release our project website at \href{https://xwang2775.github.io/horizon-leaderboard/}{HORIZON Leaderboard} and welcome contributions from the community.