MobileBench-OL: A Comprehensive Chinese Benchmark for Evaluating Mobile GUI Agents in Real-World Environment

TL;DR

MobileBench-OL is a new comprehensive Chinese benchmark designed to evaluate mobile GUI agents' task execution, reasoning, and noise robustness in real-world environments, addressing limitations of previous benchmarks.

Contribution

It introduces a large-scale, multi-dimensional online benchmark with an auto-eval framework for realistic assessment of GUI agents in Chinese mobile apps.

Findings

Significant performance gaps in current GUI agents.

MobileBench-OL effectively measures real-world capabilities.

Human evaluation confirms benchmark reliability.

Abstract

Recent advances in mobile Graphical User Interface (GUI) agents highlight the growing need for comprehensive evaluation benchmarks. While new online benchmarks offer more realistic testing than offline ones, they tend to focus on the agents' task instruction-following ability while neglecting their reasoning and exploration ability. Moreover, these benchmarks do not consider the random noise in real-world mobile environments. This leads to a gap between benchmarks and real-world environments. To addressing these limitations, we propose MobileBench-OL, an online benchmark with 1080 tasks from 80 Chinese apps. It measures task execution, complex reasoning, and noise robustness of agents by including 5 subsets, which set multiple evaluation dimensions. We also provide an auto-eval framework with a reset mechanism, enabling stable and repeatable real-world benchmarking. Evaluating 12 leading GUI agents on MobileBench-OL shows significant room for improvement to meet real-world requirements. Human evaluation further confirms that MobileBench-OL can reliably measure the performance of leading GUI agents in real environments. Our data and code will be released upon acceptance.