AgentEscapeBench: Evaluating Out-of-Domain Tool-Grounded Reasoning in LLM Agents

TL;DR

AgentEscapeBench is a benchmark designed to evaluate the ability of LLM agents to perform complex, long-range, tool-grounded reasoning tasks involving multiple dependencies and external functions.

Contribution

The paper introduces a novel escape-room-style benchmark with automated evaluation to measure LLM agents' performance on out-of-domain, long-range reasoning with external tools.

Findings

Performance drops significantly with increased dependency depth for both humans and models.

Humans maintain higher success rates than models at all difficulty levels.

Failures are mainly due to breakdowns in long-range state tracking and intermediate-result propagation.

Abstract

As LLM-based agents increasingly rely on external tools, it is important to evaluate their ability to sustain tool-grounded reasoning beyond familiar workflows and short-range interactions. We introduce AgentEscapeBench, an escape-room-style benchmark that tests whether agents can infer, execute, and revise novel tool-use procedures under explicit long-range dependency constraints. Each task defines a directed acyclic dependency graph over tools and items, requiring agents to invoke real external functions, track hidden state revealed incrementally, propagate intermediate results, and submit a deterministically verifiable final answer. AgentEscapeBench includes 270 instances across five difficulty tiers and supports fully automated evaluation. Experiments with sixteen LLM agents and human participants show that performance drops sharply as dependency depth increases: humans decline from 98.3% success at difficulty-5 to 80.0% at difficulty-25, while the best model drops from 90.0% to 60.0%. Trajectory analysis attributes model failures mainly to breakdowns in long-range state tracking, clue adherence, and intermediate-result propagation. These findings suggest that current agents can often handle local tool use but still struggle with deep contextual dependencies. We hope AgentEscapeBench can serve as a diagnostic testbed for measuring current agent capabilities and informing future training efforts toward more robust general-purpose reasoning, action, and adaptation.