WebSynthesis: World-Model-Guided MCTS for Efficient WebUI-Trajectory Synthesis

TL;DR

WebSynthesis introduces a world-model-guided MCTS framework for efficient web trajectory synthesis, enabling scalable self-improvement of web agents by simulating web environments and reducing API costs.

Contribution

The paper presents WebSynthesis, a novel approach that uses a learned world model and tree-based planning to generate high-quality web trajectories efficiently.

Findings

Agent trained with WebSynthesis matches or exceeds real-data trained models.

Supports scalable, cost-effective web trajectory generation.

Enables reversible simulation of web environments.

Abstract

Recent advancements in large language models (LLMs) have significantly improved the capabilities of web agents. However, effectively navigating complex and dynamic web environments still requires more advanced trajectory-level planning and execution. Prior studies have addressed self-improving agents by collecting extensive GUI trajectories from real-environment interactions. Despite their effectiveness, these approaches encounter two critical challenges: (1) Uncontrollable environment states, where real or sandboxed web environments often yield unstable and non-deterministic feedback, complicating the reproduction and debugging of agent behaviors; and (2) High API costs, as generating even a single interaction trajectory can involve hundreds of queries, leading to considerable API usage and computational expenses. To address these limitations and enable scalable self-improvement for agents, we propose WebSynthesis, a novel framework for trajectory synthesis and training. WebSynthesis leverages a learned world model to simulate virtual web environments, allowing a policy agent to perform efficient and reversible tree-based planning. This approach supports the large-scale generation of diverse and high-quality trajectories, which are subsequently utilized to refine the agent's policy. Experimental results demonstrate that an agent trained using WebSynthesis on a small-scale synthetic dataset achieves performance comparable to or even surpassing that of models trained on large-scale real-world data.