Step-GUI Technical Report

TL;DR

This paper presents a self-evolving training pipeline for GUI automation models, achieving high accuracy at low cost, introducing a hierarchical protocol for privacy-preserving execution, and establishing a real-world benchmark for mobile usage scenarios.

Contribution

It introduces the Calibrated Step Reward System for efficient training, the Step-GUI models with state-of-the-art performance, and the GUI-MCP protocol for privacy-aware deployment, along with a new benchmark AndroidDaily.

Findings

Achieved >90% annotation accuracy with 10-100x lower cost.

State-of-the-art GUI performance on multiple benchmarks.

Demonstrated practical deployment potential with privacy-preserving protocols.

Abstract

Recent advances in multimodal large language models unlock unprecedented opportunities for GUI automation. However, a fundamental challenge remains: how to efficiently acquire high-quality training data while maintaining annotation reliability? We introduce a self-evolving training pipeline powered by the Calibrated Step Reward System, which converts model-generated trajectories into reliable training signals through trajectory-level calibration, achieving >90% annotation accuracy with 10-100x lower cost. Leveraging this pipeline, we introduce Step-GUI, a family of models (4B/8B) that achieves state-of-the-art GUI performance (8B: 80.2% AndroidWorld, 48.5% OSWorld, 62.6% ScreenShot-Pro) while maintaining robust general capabilities. As GUI agent capabilities improve, practical deployment demands standardized interfaces across heterogeneous devices while protecting user privacy. To this end, we propose GUI-MCP, the first Model Context Protocol for GUI automation with hierarchical architecture that combines low-level atomic operations and high-level task delegation to local specialist models, enabling high-privacy execution where sensitive data stays on-device. Finally, to assess whether agents can handle authentic everyday usage, we introduce AndroidDaily, a benchmark grounded in real-world mobile usage patterns with 3146 static actions and 235 end-to-end tasks across high-frequency daily scenarios (8B: static 89.91%, end-to-end 52.50%). Our work advances the development of practical GUI agents and demonstrates strong potential for real-world deployment in everyday digital interactions.