WebForge: Breaking the Realism-Reproducibility-Scalability Trilemma in Browser Agent Benchmark

TL;DR

WebForge introduces an automated framework for creating realistic, reproducible, and scalable browser agent benchmarks with multi-dimensional capability profiling.

Contribution

It presents the first fully automated pipeline that overcomes the realism-reproducibility-scalability trilemma in browser benchmarking.

Findings

WebForge-Bench includes 934 tasks across 7 domains and 3 difficulty levels.

Difficulty stratification effectively differentiates model capabilities.

Multi-dimensional evaluation reveals capability biases invisible to aggregate metrics.

Abstract

Existing browser agent benchmarks face a fundamental trilemma: real-website benchmarks lack reproducibility due to content drift, controlled environments sacrifice realism by omitting real-web noise, and both require costly manual curation that limits scalability. We present WebForge, the first fully automated framework that resolves this trilemma through a four-agent pipeline -- Plan, Generate, Refine, and Validate -- that produces interactive, self-contained web environments end-to-end without human annotation. A seven-dimensional difficulty control framework structures task design along navigation depth, visual complexity, reasoning difficulty, and more, enabling systematic capability profiling beyond single aggregate scores. Using WebForge, we construct WebForge-Bench, a benchmark of 934 tasks spanning 7 domains and 3 difficulty levels. Multi-model experiments show that difficulty stratification effectively differentiates model capabilities, while cross-domain analysis exposes capability biases invisible to aggregate metrics. Together, these results confirm that multi-dimensional evaluation reveals distinct capability profiles that a single aggregate score cannot capture. Code and benchmark are publicly available at https://github.com/yuandaxia2001/WebForge.