Phi-Ground Tech Report: Advancing Perception in GUI Grounding

TL;DR

This paper presents Phi-Ground, a new model family that significantly improves GUI grounding accuracy for multimodal reasoning agents, achieving state-of-the-art results on multiple benchmarks and providing insights into training and data collection.

Contribution

The paper introduces Phi-Ground, a novel grounding model family that sets new performance standards across five benchmarks and offers detailed empirical analysis of training procedures.

Findings

Achieves SOTA scores of 43.2 on ScreenSpot-pro and 27.2 on UI-Vision.

Provides comprehensive empirical study on data collection and training.

Demonstrates the effectiveness of design choices in grounding models.

Abstract

With the development of multimodal reasoning models, Computer Use Agents (CUAs), akin to Jarvis from \textit{"Iron Man"}, are becoming a reality. GUI grounding is a core component for CUAs to execute actual actions, similar to mechanical control in robotics, and it directly leads to the success or failure of the system. It determines actions such as clicking and typing, as well as related parameters like the coordinates for clicks. Current end-to-end grounding models still achieve less than 65\% accuracy on challenging benchmarks like ScreenSpot-pro and UI-Vision, indicating they are far from being ready for deployment. % , as a single misclick can result in unacceptable consequences. In this work, we conduct an empirical study on the training of grounding models, examining details from data collection to model training. Ultimately, we developed the \textbf{Phi-Ground} model family, which achieves state-of-the-art performance across all five grounding benchmarks for models under $10B$ parameters in agent settings. In the end-to-end model setting, our model still achieves SOTA results with scores of \textit{\textbf{43.2}} on ScreenSpot-pro and \textit{\textbf{27.2}} on UI-Vision. We believe that the various details discussed in this paper, along with our successes and failures, not only clarify the construction of grounding models but also benefit other perception tasks. Project homepage: \href{https://zhangmiaosen2000.github.io/Phi-Ground/}{https://zhangmiaosen2000.github.io/Phi-Ground/}