The Amazing Agent Race: Strong Tool Users, Weak Navigators

TL;DR

The paper introduces The Amazing Agent Race (AAR), a new benchmark with DAG puzzles to evaluate LLM agents' navigation and tool-use abilities, revealing significant navigation failures in current models.

Contribution

It presents a novel DAG-based benchmark with diverse difficulty levels and validation, highlighting navigation as a key challenge for LLM agents beyond tool execution.

Findings

Best agent achieves only 37.2% accuracy on AAR.

Navigation errors account for 27-52% of failures.

Agent architecture and scale significantly impact performance.

Abstract

Existing tool-use benchmarks for LLM agents are overwhelmingly linear: our analysis of six benchmarks shows 55 to 100% of instances are simple chains of 2 to 5 steps. We introduce The Amazing Agent Race (AAR), a benchmark featuring directed acyclic graph (DAG) puzzles (or "legs") with fork-merge tool chains. We release 1,400 instances across two variants: sequential (800 legs) and compositional (600 DAG legs). Agents must navigate Wikipedia, execute multi-step tool chains, and aggregate results into a verifiable answer. Legs are procedurally generated from Wikipedia seeds across four difficulty levels with live-API validation. Three complementary metrics (finish-line accuracy, pit-stop visit rate, and roadblock completion rate) separately diagnose navigation, tool-use, and arithmetic failures. Evaluating three agent frameworks on 1,400 legs, the best achieves only 37.2% accuracy. Navigation errors dominate (27 to 52% of trials) while tool-use errors remain below 17%, and agent architecture matters as much as model scale (Claude Code matches Codex CLI at 37% with 6x fewer tokens). The compositional structure of AAR reveals that agents fail not at calling tools but at navigating to the right pages, a blind spot invisible to linear benchmarks. The project page can be accessed at: https://minnesotanlp.github.io/the-amazing-agent-race