Toward Autonomous UI Exploration: The UIExplorer Benchmark

TL;DR

This paper introduces UIExplore-Bench, a benchmark for evaluating autonomous agents' ability to explore user interfaces, providing standardized tasks, metrics, and results that highlight current limitations and future research directions.

Contribution

We present the first dedicated benchmark for UI exploration, formalize exploration metrics, and evaluate agents, setting a foundation for future advancements in autonomous UI understanding.

Findings

UIExplore-AlGo achieves up to 77.2% of human performance in Structured mode.

Agents perform significantly below human experts, indicating room for improvement.

Benchmark and dataset are publicly released to foster further research.

Abstract

Autonomous agents must know how to explore user interfaces (UIs) for reliable task solving, yet systematic evaluation of this crucial phase is lacking. We introduce UIExplore-Bench, the first benchmark explicitly dedicated to UI exploration. The benchmark evaluates agents with either Structured mode (granting access to layout information like DOM trees) or Screen mode (relying on GUI-only observations such as screenshots and human-like mouse/keyboard interactions) across three levels in a standardized GitLab sandbox environment. We formalize exploration as the process of maximizing the set of actionable UI components discovered and propose a metric, human-normalized UI-Functionalities Observed (hUFO), to quantify the effectiveness of exploration. Our results show that UIExplore-AlGo achieves the leading mean hUFO scores, reaching up to 77.2% of human performance in Structured mode and 59.0% in Screen mode at 2,000 steps, particularly excelling at the Sparse level. The results highlight the relevance of our benchmark, as current agents show a substantial performance gap compared to one hour of human expert exploration, indicating ample room for future advancements. We publicly release the benchmark environment, an exploration dataset, and an evaluation suite to catalyze research into efficient UI exploration strategies and their downstream applications, such as experience-driven task completion and automated training data generation.