PATHWAYS: Evaluating Investigation and Context Discovery in AI Web Agents

TL;DR

PATHWAYS introduces a benchmark to evaluate web-based AI agents' ability to discover and utilize hidden context, revealing significant limitations in their investigative reasoning and evidence integration capabilities.

Contribution

The paper presents a new benchmark with 250 decision tasks to systematically assess and highlight current deficiencies in AI web agents' investigative and contextual reasoning skills.

Findings

Agents often find relevant pages but rarely access decisive evidence.

Performance drops sharply when tasks involve misleading signals.

Agents frequently hallucinate reasoning and fail to integrate discovered context.

Abstract

We introduce PATHWAYS, a benchmark of 250 multi-step decision tasks that test whether web-based agents can discover and correctly use hidden contextual information. Across both closed and open models, agents typically navigate to relevant pages but retrieve decisive hidden evidence in only a small fraction of cases. When tasks require overturning misleading surface-level signals, performance drops sharply to near chance accuracy. Agents frequently hallucinate investigative reasoning by claiming to rely on evidence they never accessed. Even when correct context is discovered, agents often fail to integrate it into their final decision. Providing more explicit instructions improves context discovery but often reduces overall accuracy, revealing a tradeoff between procedural compliance and effective judgement. Together, these results show that current web agent architectures lack reliable mechanisms for adaptive investigation, evidence integration, and judgement override.