Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction

TL;DR

Aguvis introduces a unified vision-based framework for autonomous GUI agents that operate directly on screen images, enabling cross-platform interaction and structured reasoning, achieving state-of-the-art results without relying on closed-source models.

Contribution

The paper presents Aguvis, a novel vision-based GUI agent framework with a large-scale dataset and a two-stage training pipeline, advancing autonomous GUI interaction without textual or platform-specific dependencies.

Findings

Achieves state-of-the-art performance on offline and online benchmarks.

Operates without closed-source models, fully autonomous.

Provides open-source datasets and models for future research.

Abstract

Automating GUI tasks remains challenging due to reliance on textual representations, platform-specific action spaces, and limited reasoning capabilities. We introduce Aguvis, a unified vision-based framework for autonomous GUI agents that directly operates on screen images, standardizes cross-platform interactions and incorporates structured reasoning via inner monologue. To enable this, we construct Aguvis Data Collection, a large-scale dataset with multimodal grounding and reasoning annotations, and develop a two-stage training pipeline that separates GUI grounding from planning and reasoning. Experiments show that Aguvis achieves state-of-the-art performance across offline and real-world online benchmarks, marking the first fully autonomous vision-based GUI agent that operates without closed-source models. We open-source all datasets, models, and training recipes at https://aguvis-project.github.io to advance future research.