WebArXiv: Evaluating Multimodal Agents on Time-Invariant arXiv Tasks

TL;DR

WebArXiv introduces a static, reproducible benchmark for evaluating multimodal web agents on arXiv tasks, addressing evaluation instability and proposing a dynamic reflection mechanism to improve agent performance.

Contribution

The paper presents WebArXiv, a novel static benchmark for web agents, and introduces a dynamic reflection method to enhance agent decision-making and evaluation reliability.

Findings

WebArXiv provides consistent, reproducible evaluation of web agents.

Agents show varied performance on WebArXiv, highlighting the benchmark's discriminative power.

The dynamic reflection mechanism improves agent decision accuracy.

Abstract

Recent progress in large language models (LLMs) has enabled the development of autonomous web agents capable of navigating and interacting with real websites. However, evaluating such agents remains challenging due to the instability and inconsistency of existing benchmarks, which often rely on dynamic content or oversimplified simulations. In this work, we introduce WebArXiv, a static and time-invariant benchmark comprising 275 web-based tasks grounded in the arXiv platform. WebArXiv ensures reproducible and reliable evaluation by anchoring tasks in fixed web snapshots with deterministic ground truths and standardized action trajectories. Through behavioral analysis, we identify a common failure mode, Rigid History Reflection, where agents over-rely on fixed interaction histories. To address this, we propose a lightweight dynamic reflection mechanism that allows agents to selectively retrieve relevant past steps during decision-making. We evaluate ten state-of-the-art web agents on WebArXiv. Results demonstrate clear performance differences across agents and validate the effectiveness of our proposed reflection strategy.