AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs

TL;DR

AutoGUI introduces a scalable pipeline leveraging large language models to automatically generate detailed UI element functionality annotations, significantly improving UI grounding and enabling advanced software automation tasks.

Contribution

The paper presents a novel LLM-based method for large-scale, high-quality UI annotation, creating the AutoGUI-704k dataset with detailed functionality labels.

Findings

AutoGUI-704k dataset improves UI grounding accuracy.

Annotations are comparable to human quality.

Enhanced performance in UI agent tasks.

Abstract

User interface understanding with vision-language models (VLMs) has received much attention due to its potential for enhancing software automation. However, existing datasets used to build UI-VLMs either only contain large-scale context-free element annotations or contextualized functional descriptions for elements at a small scale. In this work, we propose the \textbf{AutoGUI} pipeline for automatically annotating UI elements with detailed functionality descriptions at scale. Specifically, we leverage large language models (LLMs) to infer element functionality by comparing UI state changes before and after simulated interactions. To improve annotation quality, we propose LLM-aided rejection and verification, eliminating invalid annotations without human labor. We construct a high-quality AutoGUI-704k dataset using the proposed pipeline, featuring diverse and detailed functionality annotations that are hardly provided by previous datasets. Human evaluation shows that we achieve annotation correctness comparable to a trained human annotator. Extensive experiments show that our dataset remarkably enhances VLM's UI grounding capabilities and exhibits significant scaling effects. We also show the interesting potential use of our dataset in UI agent tasks. Please view our project at https://autogui-project.github.io/.