AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials

TL;DR

AgentTrek introduces a scalable pipeline that synthesizes web agent trajectories from tutorials, enabling cost-effective training of GUI agents with multimodal data and achieving state-of-the-art results.

Contribution

We propose a novel automated method to generate high-quality web agent trajectories from tutorials, reducing reliance on manual annotation and enabling scalable training.

Findings

Achieves state-of-the-art performance on web browsing benchmarks

Reduces data collection cost to $0.55 per trajectory

Demonstrates effective multimodal, guided replay for agent training

Abstract

Graphical User Interface (GUI) agents can automate complex tasks across digital environments, but their development is hindered by the scarcity of high-quality trajectory data for training. Existing approaches rely on expensive human annotation, making them unsustainable at scale. We propose AgentTrek, a scalable data synthesis pipeline that generates web agent trajectories by leveraging publicly available tutorials. Our three-stage method: (1) automatically harvests and filters tutorial-like texts from the internet using a specialized classification model, (2) transforms these texts into structured task specifications with step-by-step instructions, and (3) employs a visual-language model (VLM) agent to execute these instructions in real environments, while a VLM-based evaluator verifies trajectory correctness. The synthesized trajectories encompass multiple modalities, including text-based HTML observations with function-calling API actions, and vision-based screenshot observations with pixel-level actions. This multimodal data, enriched with chain-of-thought reasoning, enables agents to achieve state-of-the-art performance on both textual web browsing benchmarks (e.g., WebArena) and visual web grounding and browsing benchmarks (e.g., ScreenSpot Web and Multimodal Mind2Web). Furthermore, our fully automated approach significantly reduces data collection costs, achieving a cost of just $0.55 per high-quality trajectory without human annotators. Our work demonstrates that guided replay using web tutorials is a practical and scalable strategy for training advanced GUI agents, paving the way for more capable and autonomous digital assistants.