AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web

TL;DR

AgentWebBench is a new benchmark for evaluating multi-agent coordination in the emerging Agentic Web paradigm, analyzing web information synthesis and interaction strategies across various models and tasks.

Contribution

It introduces a comprehensive benchmark for multi-agent web interaction, evaluates multiple models and strategies, and provides insights into the properties and challenges of decentralized web information access.

Findings

Multi-agent coordination lags behind centralized retrieval but improves with model scale.

On question answering, multi-agent approaches can outperform centralized retrieval.

Decentralized access concentrates traffic and benefits from better planning and interaction scaling.

Abstract

Agentic Web is an emerging paradigm where autonomous agents help users use online information. As the paradigm develops, content providers are also deploying agents to manage their data and serve it through controlled interfaces. This shift moves information access from centralized retrieval to decentralized coordination. To study this setting, we introduce AgentWebBench, a benchmark that evaluates how well a user agent synthesizes answers by interacting with website-specific content agents. We evaluate four tasks that cover common web information needs, spanning ranked retrieval (web search, web recommendation) and open-ended synthesis (question answering, deep research). Across seven advanced LLMs and three coordination strategies, multi-agent coordination generally lags behind centralized retrieval as expected, because user agent cannot directly access the corpus, but the gap shrinks with model scale and can even outperform centralized retrieval on question answering. This benchmark also enables us to study properties of the emerging paradigm of the digital world. We find that decentralized access concentrates traffic toward a small set of websites, test time scaling improves both interaction reliability and task performance, and strong results require sufficient interactions guided by careful planning. Finally, our failure analysis suggests that user agents need better planning and answer synthesis, while content agents need more reliable retrieval and evidence quality. Code, data, and APIs are released on https://github.com/cxcscmu/AgentWebBench.