WebArbiter: A Principle-Guided Reasoning Process Reward Model for Web Agents

TL;DR

WebArbiter introduces a principle-guided, reasoning-based reward model for web agents that generates structured justifications, improving task success and robustness over existing methods.

Contribution

It presents WebArbiter, a novel WebPRM that uses text generation for reasoning and verdicts, with a two-stage training pipeline and a new benchmark for evaluation.

Findings

WebArbiter-7B outperforms GPT-5 by 9.1 points on WebPRMBench.

It surpasses prior WebPRMs by up to 6.4 points in trajectory search.

The model demonstrates improved robustness and interpretability in complex web tasks.

Abstract

Web agents hold great potential for automating complex computer tasks, yet their interactions involve long-horizon, sequential decision-making with irreversible actions. In such settings, outcome-based supervision is sparse and delayed, often rewarding incorrect trajectories and failing to support inference-time scaling. This motivates the use of Process Reward Models (WebPRMs) for web navigation, but existing approaches remain limited: scalar WebPRMs collapse progress into coarse, weakly grounded signals, while checklist-based WebPRMs rely on brittle template matching that fails under layout or semantic changes and often mislabels superficially correct actions as successful, providing little insight or interpretability. To address these challenges, we introduce WebArbiter, a reasoning-first, principle-inducing WebPRM that formulates reward modeling as text generation, producing structured justifications that conclude with a preference verdict and identify the action most conducive to task completion under the current context. Training follows a two-stage pipeline: reasoning distillation equips the model with coherent principle-guided reasoning, and reinforcement learning corrects teacher biases by directly aligning verdicts with correctness, enabling stronger generalization. To support systematic evaluation, we release WebPRMBench, a comprehensive benchmark spanning four diverse web environments with rich tasks and high-quality preference annotations. On WebPRMBench, WebArbiter-7B outperforms the strongest baseline, GPT-5, by 9.1 points. In reward-guided trajectory search on WebArena-Lite, it surpasses the best prior WebPRM by up to 6.4 points, underscoring its robustness and practical value in complex web tasks.