Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents

TL;DR

This paper introduces WebDreamer, a model-based planning framework using LLMs as world models for web agents, improving efficiency and performance in real-world web tasks compared to reactive and search-based methods.

Contribution

It proposes a novel framework employing LLMs as world models for web agents and demonstrates scalable training of specialized models for improved planning efficiency.

Findings

WebDreamer outperforms reactive baselines in web tasks.

It is 4-5 times more efficient than tree search in sandbox environments.

Dreamer-7B performs comparably to GPT-4o in web planning.

Abstract

Language agents based on large language models (LLMs) have demonstrated great promise in automating web-based tasks. Recent work has shown that incorporating advanced planning algorithms, e.g., tree search, is advantageous over reactive planning for web agents. However, unlike simulated sandbox environments, real-world environments such as the web are rife with irreversible actions. This undermines the feasibility of backtracking, a cornerstone of (tree) search. Overly relying on test-time search also hurts efficiency. We advocate model-based planning for web agents that employs a world model to simulate and deliberate over the outcome of each candidate action before committing to one. We systematically explore this paradigm by (1) Proposing a model-based planning framework, WebDreamer, which employs LLMs to serve as both world models and value functions; (2) Training specialized LLMs as world models with a scalable data synthesis pipeline. Empirical results demonstrate that WebDreamer achieves substantial performance improvements over reactive baselines. It is competitive, while being 4-5 times more efficient, with tree search in sandbox environments (VisualWebArena) and also works effectively on real-world websites (Online-Mind2Web and Mind2Web-Live). Furthermore, our trained world model, Dreamer-7B, performs comparable to GPT-4o, highlighting the potential of specialized world models for efficient and effective planning in complex web environments.