D-GARA: A Dynamic Benchmarking Framework for GUI Agent Robustness in Real-World Anomalies

TL;DR

D-GARA is a dynamic benchmarking framework designed to evaluate Android GUI agent robustness against real-world anomalies, revealing significant performance drops and emphasizing the need for robustness-aware learning.

Contribution

We introduce D-GARA, a novel extensible framework and benchmark for testing GUI agent robustness in realistic anomaly scenarios, addressing limitations of static, idealized datasets.

Findings

State-of-the-art GUI agents show performance degradation with anomalies.

D-GARA supports diverse, real-world anomaly types for comprehensive evaluation.

Framework's modular design allows easy extension and customization.

Abstract

Developing intelligent agents capable of operating a wide range of Graphical User Interfaces (GUIs) with human-level proficiency is a key milestone on the path toward Artificial General Intelligence. While most existing datasets and benchmarks for training and evaluating GUI agents are static and idealized, failing to reflect the complexity and unpredictability of real-world environments, particularly the presence of anomalies. To bridge this research gap, we propose D-GARA, a dynamic benchmarking framework, to evaluate Android GUI agent robustness in real-world anomalies. D-GARA introduces a diverse set of real-world anomalies that GUI agents commonly face in practice, including interruptions such as permission dialogs, battery warnings, and update prompts. Based on D-GARA framework, we construct and annotate a benchmark featuring commonly used Android applications with embedded anomalies to support broader community research. Comprehensive experiments and results demonstrate substantial performance degradation in state-of-the-art GUI agents when exposed to anomaly-rich environments, highlighting the need for robustness-aware learning. D-GARA is modular and extensible, supporting the seamless integration of new tasks, anomaly types, and interaction scenarios to meet specific evaluation goals.