SecAgent: Efficient Mobile GUI Agent with Semantic Context

TL;DR

SecAgent is an efficient 3B-scale mobile GUI agent that leverages a new Chinese dataset and semantic context summarization to improve task automation while reducing computational costs.

Contribution

The paper introduces a Chinese mobile GUI dataset, a semantic context mechanism, and demonstrates that SecAgent outperforms similar-scale models on navigation benchmarks.

Findings

SecAgent achieves performance comparable to larger models (7B-8B) on navigation tasks.

The semantic context mechanism reduces computational costs significantly.

The dataset includes 18k grounding samples and 121k navigation steps across 44 apps.

Abstract

Mobile Graphical User Interface (GUI) agents powered by multimodal large language models have demonstrated promising capabilities in automating complex smartphone tasks. However, existing approaches face two critical limitations: the scarcity of high-quality multilingual datasets, particularly for non-English ecosystems, and inefficient history representation methods. To address these challenges, we present SecAgent, an efficient mobile GUI agent at 3B scale. We first construct a human-verified Chinese mobile GUI dataset with 18k grounding samples and 121k navigation steps across 44 applications, along with a Chinese navigation benchmark featuring multi-choice action annotations. Building upon this dataset, we propose a semantic context mechanism that distills history screenshots and actions into concise, natural language summaries, significantly reducing computational costs while preserving task-relevant information. Through supervised and reinforcement fine-tuning, SecAgent outperforms similar-scale baselines and achieves performance comparable to 7B-8B models on our and public navigation benchmarks. Our dataset is available at https://huggingface.co/datasets/alibabagroup/CMGUI.