GUI-360$^\circ$: A Comprehensive Dataset and Benchmark for Computer-Using Agents

TL;DR

GUI-360$^ circ$ is a large-scale dataset and benchmark suite for developing and evaluating computer-using agents, addressing key gaps with automated data collection, multi-modal annotations, and comprehensive task evaluation.

Contribution

The paper introduces GUI-360$^ circ$, a novel large-scale dataset and benchmark with automated pipelines for multi-modal GUI tasks, enabling progress in computer-using agents.

Findings

State-of-the-art models show significant gaps in GUI grounding and action prediction.

Supervised fine-tuning and reinforcement learning improve model performance.

Current models still fall short of human-level reliability.

Abstract

We introduce GUI-360$^\circ$, a large-scale, comprehensive dataset and benchmark suite designed to advance computer-using agents (CUAs). CUAs present unique challenges and is constrained by three persistent gaps: a scarcity of real-world CUA tasks, the lack of automated collection-and-annotation pipelines for multi-modal trajectories, and the absence of a unified benchmark that jointly evaluates GUI grounding, screen parsing, and action prediction. GUI-360$^\circ$ addresses these gaps with an LLM-augmented, largely automated pipeline for query sourcing, environment-template construction, task instantiation, batched execution, and LLM-driven quality filtering. The released corpus contains over 1.2M executed action steps across thousands of trajectories in popular Windows office applications, and includes full-resolution screenshots, accessibility metadata when available, instantiated goals, intermediate reasoning traces, and both successful and failed action trajectories. The dataset supports three canonical tasks, GUI grounding, screen parsing, and action prediction, and a hybrid GUI+API action space that reflects modern agent designs. Benchmarking state-of-the-art vision--language models on GUI-360$^\circ$ reveals substantial out-of-the-box shortcomings in grounding and action prediction; supervised fine-tuning and reinforcement learning yield significant gains but do not close the gap to human-level reliability. We release GUI-360$^\circ$ and accompanying code to facilitate reproducible research and accelerate progress on robust desktop CUAs. The full dataset has been made public on https://huggingface.co/datasets/vyokky/GUI-360.