Evaluating Long-Context Reasoning in LLM-Based WebAgents

TL;DR

This paper evaluates the ability of large language model-based WebAgents to reason over extended interaction histories, revealing significant performance drops in long contexts and highlighting key challenges for real-world deployment.

Contribution

It introduces a benchmark and evaluation framework for long context reasoning in WebAgents, and provides empirical insights into model limitations and potential improvements.

Findings

Performance drops from 40-50% to less than 10% in long contexts

Agents often get stuck in loops and lose track of tasks

Implicit RAG offers modest improvements in summarization

Abstract

As large language model (LLM)-based agents become increasingly integrated into daily digital interactions, their ability to reason across long interaction histories becomes crucial for providing personalized and contextually aware assistance. However, the performance of these agents in long context scenarios, particularly for action-taking WebAgents operating in realistic web environments, remains largely unexplored. This paper introduces a benchmark for evaluating long context reasoning capabilities of WebAgents through sequentially dependent subtasks that require retrieval and application of information from extended interaction histories. We develop a novel evaluation framework that simulates multi-session user interactions by injecting irrelevant task trajectories between dependent subtasks, creating contexts ranging from 25,000 to 150,000 tokens. Through extensive evaluation of four popular models, Claude-3.7, GPT-4.1, Llama 4, and o4-mini, we observe a dramatic performance degradation as context length increases, with success rates dropping from 40-50\% in baseline conditions to less than 10\% in long context scenarios. Our detailed error analysis reveals that agents primarily fail due to getting stuck in loops and losing track of original task objectives. We further propose an implicit RAG approach that provides modest improvements by generating task-relevant summaries, though fundamental limitations in long context reasoning persist. These findings highlight critical challenges for deploying WebAgents in realistic, long-term user interaction scenarios and provide insights for developing more robust agent architectures capable of maintaining coherent task execution across extended contexts.