GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents

TL;DR

GUI-CEval is a comprehensive Chinese benchmark for mobile GUI agents, evaluating perception, planning, reflection, execution, and evaluation across diverse apps and device types to improve model reliability.

Contribution

This paper introduces GUI-CEval, the first detailed Chinese mobile GUI benchmark built on physical devices, covering full capability assessment from perception to execution.

Findings

Models like Qwen2.5-VL and UI-TARS perform well

Most MLLMs struggle with reflective decision-making

Weaknesses in self-evaluation limit real-world reliability

Abstract

Recent progress in Multimodal Large Language Models (MLLMs) has enabled mobile GUI agents capable of visual perception, cross-modal reasoning, and interactive control. However, existing benchmarks are largely English-centric and fail to capture the linguistic and interaction characteristics of the Chinese mobile ecosystem. They also focus on isolated skills such as GUI grounding or offline agent, lacking a unified and fine-grained framework to assess the full capability chain from perception to execution. To address this gap, we introduce GUI-CEval, the first comprehensive benchmark for Chinese mobile GUI agents, built entirely on physical device environments. GUI-CEval spans 201 mainstream apps across four device types and adopts a two-level structure that evaluates both atomic abilities and realistic application-level performance along five dimensions: perception, planning, reflection, execution, and evaluation. All data are collected and verified through multi-stage manual processes to ensure authenticity and reproducibility. Extensive experiments on 20 representative MLLMs and multi-agent systems show that while models such as Qwen2.5-VL and UI-TARS perform competitively, most MLLMs still exhibit clear weaknesses in reflective decision-making and post-action self-evaluation, limiting their reliability in real-world interactions. We hope GUI-CEval provides a comprehensive and interpretable benchmark to guide capability diagnosis and advance the development of Chinese mobile GUI agents.