AUTO-Explorer: Automated Data Collection for GUI Agent

TL;DR

Auto-Explorer introduces an automated, low-cost data collection method for GUI agents that efficiently explores GUI environments, enabling rapid fine-tuning of multimodal large language models for software interaction tasks.

Contribution

It presents a novel exploration mechanism for GUI data collection, along with the UIXplore benchmark to evaluate exploration quality, improving model adaptation to new software environments.

Findings

Auto-Explorer outperforms existing methods in data collection efficiency.

Fine-tuning with Auto-Explorer data enhances MLLM performance on GUI tasks.

The UIXplore benchmark effectively measures exploration strategy quality.

Abstract

Recent advancements in GUI agents have significantly expanded their ability to interpret natural language commands to manage software interfaces. However, acquiring GUI data remains a significant challenge. Existing methods often involve designing automated agents that browse URLs from the Common Crawl, using webpage HTML to collect screenshots and corresponding annotations, including the names and bounding boxes of UI elements. However, this method is difficult to apply to desktop software or some newly launched websites not included in the Common Crawl. While we expect the model to possess strong generalization capabilities to handle this, it is still crucial for personalized scenarios that require rapid and perfect adaptation to new software or websites. To address this, we propose an automated data collection method with minimal annotation costs, named Auto-Explorer. It incorporates a simple yet effective exploration mechanism that autonomously parses and explores GUI environments, gathering data efficiently. Additionally, to assess the quality of exploration, we have developed the UIXplore benchmark. This benchmark creates environments for explorer agents to discover and save software states. Using the data gathered, we fine-tune a multimodal large language model (MLLM) and establish a GUI element grounding testing set to evaluate the effectiveness of the exploration strategies. Our experiments demonstrate the superior performance of Auto-Explorer, showing that our method can quickly enhance the capabilities of an MLLM in explored software.