OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Models

TL;DR

OS-Oracle introduces a comprehensive framework with a new dataset, training paradigm, and benchmark for developing and evaluating cross-platform GUI critic models, significantly advancing step-level decision-making in GUI navigation agents.

Contribution

The paper presents a scalable data pipeline, a novel two-stage training method, and a holistic benchmark for GUI critic models, addressing data scarcity and evaluation challenges.

Findings

OS-Oracle-7B achieves state-of-the-art results on OS-Critic Bench.

The critic model surpasses proprietary models in mobile domain.

Pre-critic use of OS-Oracle-7B improves native GUI agent performance.

Abstract

With VLM-powered computer-using agents (CUAs) becoming increasingly capable at graphical user interface (GUI) navigation and manipulation, reliable step-level decision-making has emerged as a key bottleneck for real-world deployment. In long-horizon workflows, errors accumulate quickly and irreversible actions can cause unintended consequences, motivating critic models that assess each action before execution. While critic models offer a promising solution, their effectiveness is hindered by the lack of diverse, high-quality GUI feedback data and public critic benchmarks for step-level evaluation in computer use. To bridge these gaps, we introduce OS-Oracle that makes three core contributions: (1) a scalable data pipeline for synthesizing cross-platform GUI critic data; (2) a two-stage training paradigm combining supervised fine-tuning (SFT) and consistency-preserving group relative policy optimization (CP-GRPO); (3) OS-Critic Bench, a holistic benchmark for evaluating critic model performance across Mobile, Web, and Desktop platforms. Leveraging this framework, we curate a high-quality dataset containing 310k critic samples. The resulting critic model, OS-Oracle-7B, achieves state-of-the-art performance among open-source VLMs on OS-Critic Bench, and surpasses proprietary models on the mobile domain. Furthermore, when serving as a pre-critic, OS-Oracle-7B improves the performance of native GUI agents such as UI-TARS-1.5-7B in OSWorld and AndroidWorld environments. The code is open-sourced at https://github.com/numbmelon/OS-Oracle.