GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomalies

TL;DR

GUI-Robust introduces a new dataset with real-world anomalies for testing GUI agents, along with a semi-automated annotation method that greatly reduces labeling effort, revealing significant robustness issues in current models.

Contribution

We present GUI-Robust, a comprehensive dataset with anomalies for GUI agent evaluation, and a semi-automated annotation paradigm that accelerates dataset creation.

Findings

State-of-the-art GUI agents perform poorly on abnormal scenarios.

The dataset reveals significant robustness gaps in current GUI agents.

Our annotation method reduces labeling time by over 19 times.

Abstract

The development of high-quality datasets is crucial for benchmarking and advancing research in Graphical User Interface (GUI) agents. Despite their importance, existing datasets are often constructed under idealized conditions, overlooking the diverse anomalies frequently encountered in real-world deployments. To address this limitation, we introduce GUI-Robust, a novel dataset designed for comprehensive GUI agent evaluation, explicitly incorporating seven common types of anomalies observed in everyday GUI interactions. Furthermore, we propose a semi-automated dataset construction paradigm that collects user action sequences from natural interactions via RPA tools and then generate corresponding step and task descriptions for these actions with the assistance of MLLMs. This paradigm significantly reduces annotation time cost by a factor of over 19 times. Finally, we assess state-of-the-art GUI agents using the GUI-Robust dataset, revealing their substantial performance degradation in abnormal scenarios. We anticipate that our work will highlight the importance of robustness in GUI agents and inspires more future research in this direction. The dataset and code are available at https://github.com/chessbean1/GUI-Robust..