Covering Human Action Space for Computer Use: Data Synthesis and Benchmark

TL;DR

This paper introduces CUActSpot, a comprehensive benchmark and data synthesis pipeline for evaluating and improving computer-use agents' ability to handle complex, diverse GUI interactions across multiple modalities.

Contribution

It presents a new benchmark, CUActSpot, and a renderer-based data synthesis pipeline to enhance model training on complex GUI interactions beyond prior click-centric datasets.

Findings

Phi-Ground-Any-4B outperforms open-source models with fewer than 32B parameters.

The benchmark covers five modalities and various actions, broadening interaction types.

The data synthesis pipeline automatically generates scenes and instructions for training.

Abstract

Computer-use agents (CUAs) automate on-screen work, as illustrated by GPT-5.4 and Claude. Yet their reliability on complex, low-frequency interactions is still poor, limiting user trust. Our analysis of failure cases from advanced models suggests a long-tail pattern in GUI operations, where a relatively small fraction of complex and diverse interactions accounts for a disproportionate share of task failures. We hypothesize that this issue largely stems from the scarcity of data for complex interactions. To address this problem, we propose a new benchmark CUActSpot for evaluating models' capabilities on complex interactions across five modalities: GUI, text, table, canvas, and natural image, as well as a variety of actions (click, drag, draw, etc.), covering a broader range of interaction types than prior click-centric benchmarks that focus mainly on GUI widgets. We also design a renderer-based data-synthesis pipeline: scenes are automatically generated for each modality, screenshots and element coordinates are recorded, and an LLM produces matching instructions and action traces. After training on this corpus, our Phi-Ground-Any-4B outperforms open-source models with fewer than 32B parameters. We will release our benchmark, data, code, and models at https://github.com/microsoft/Phi-Ground.git