MobiBench: Multi-Branch, Modular Benchmark for Mobile GUI Agents

TL;DR

MobiBench is a novel modular offline benchmarking framework for mobile GUI agents that achieves high fidelity, scalability, and reproducibility, enabling detailed component analysis and better evaluation practices.

Contribution

It introduces the first multi-path aware, modular offline benchmark for mobile GUI agents, addressing limitations of existing single-path and monolithic evaluation methods.

Findings

MobiBench achieves 94.72% agreement with human evaluators.

It enables detailed module-level analysis of mobile GUI agents.

The framework uncovers key insights into component contributions and limitations.

Abstract

Mobile GUI Agents, AI agents capable of interacting with mobile applications on behalf of users, have the potential to transform human computer interaction. However, current evaluation practices for GUI agents face two fundamental limitations. First, they either rely on single path offline benchmarks or online live benchmarks. Offline benchmarks using static, single path annotated datasets unfairly penalize valid alternative actions, while online benchmarks suffer from poor scalability and reproducibility due to the dynamic and unpredictable nature of live evaluation. Second, existing benchmarks treat agents as monolithic black boxes, overlooking the contributions of individual components, which often leads to unfair comparisons or obscures key performance bottlenecks. To address these limitations, we present MobiBench, the first modular and multi path aware offline benchmarking framework for mobile GUI agents that enables high fidelity, scalable, and reproducible evaluation entirely in offline settings. Our experiments demonstrate that MobiBench achieves 94.72 percent agreement with human evaluators, on par with carefully engineered online benchmarks, while preserving the scalability and reproducibility of static offline benchmarks. Furthermore, our comprehensive module level analysis uncovers several key insights, including a systematic evaluation of diverse techniques used in mobile GUI agents, optimal module configurations across model scales, the inherent limitations of current LFMs, and actionable guidelines for designing more capable and cost efficient mobile agents.