MobiAgent: A Systematic Framework for Customizable Mobile Agents

TL;DR

MobiAgent is a comprehensive framework that enhances mobile agents with improved accuracy and efficiency through specialized models, acceleration techniques, benchmarking, and an AI-assisted data collection pipeline, achieving state-of-the-art results.

Contribution

The paper introduces MobiAgent, a systematic and customizable mobile agent system with novel components and an efficient data collection pipeline, advancing real-world mobile agent performance.

Findings

Achieves state-of-the-art performance in real-world scenarios.

Reduces manual annotation costs significantly.

Outperforms general-purpose LLMs and specialized GUI agents.

Abstract

With the rapid advancement of Vision-Language Models (VLMs), GUI-based mobile agents have emerged as a key development direction for intelligent mobile systems. However, existing agent models continue to face significant challenges in real-world task execution, particularly in terms of accuracy and efficiency. To address these limitations, we propose MobiAgent, a comprehensive mobile agent system comprising three core components: the MobiMind-series agent models, the AgentRR acceleration framework, and the MobiFlow benchmarking suite. Furthermore, recognizing that the capabilities of current mobile agents are still limited by the availability of high-quality data, we have developed an AI-assisted agile data collection pipeline that significantly reduces the cost of manual annotation. Compared to both general-purpose LLMs and specialized GUI agent models, MobiAgent achieves state-of-the-art performance in real-world mobile scenarios.

Equations

$$R_{\mathrm{iou}}=\{\begin{array}{ll}\alpha ,& \mathrm{IoU}\left(b_{\mathrm{pred}},b_{gt}\right)>\beta ,\\ 0,& \text{otherwise}.\end{array},R_{\mathrm{center}}=\{\begin{array}{ll}1-\alpha ,& \mathrm{Center}\left(b_{\mathrm{pred}}\right)\text{is inside}{b}_{gt},\\ 0,& \text{otherwise}.\end{array},$$

$$R=R_{\mathrm{iou}}+R_{\mathrm{center}},$$

$$R=R_{\mathrm{content}}=\{\begin{array}{ll}1,& a_{\mathrm{pred}}=a_{gt},\\ 0,& \text{otherwise}.\end{array},$$

$$v_i^{(l)}=f(T_i,l)\in {\mathbb{R}}^d.$$

$${\text{Embed-Reuse}}^{(l)}(T_i,T_j)=\{\begin{array}{ll}\text{True},& \text{if }S(v_i^{(l)},v_j^{(l)})\ge {\tau }_1\\ \text{False},& \text{otherwise}\end{array}.$$

$$\mathcal{C}=\{T\in {\mathcal{T}}_{\text{hist}}\mid {\text{Embed-Reuse}}^{(l)}(T,T_c)\},$$

$${\text{Rerank-Reuse}}^{(l)}=\{\begin{array}{ll}\text{True},& \text{if }\exists T_i\in \mathcal{C}\text{ such that }g(T_i,T_c,l)\ge {\tau }_2\\ \text{False,}& \text{otherwise}\end{array}.$$

$$v^{+}=f(T_s,l),\mathrm{pre}(T_s,T_i)\ge l,$$

$$v^{-}=\left\{f(T_t,l)\mid \mathrm{pre}(T_t,T_i)<l\right\},$$

$$\mathcal{L}={\mathbb{E}}_v\left[-\ln \frac{e^{S(v,v^{+})/\tau }}{e^{S(v,v^{+})/\tau }+{\sum }_{j=1}^M{e}^{S(v,v_j^{-})/\tau }}\right],$$

Full text

1 Introduction

With the rise of Vision-Language Models (VLMs) and the growing adoption of mobile agents, we have observed that agents leveraging GUI/XML understanding have immense application potential on real mobile devices. Unlike traditional mobile intelligent assistants, which primarily rely on rule-based and intent-driven API calls, GUI/XML-based agents are capable of adapting to any mobile application without requiring additional adaptation from app developers. As a result, an increasing number of vendors have introduced their own agent models for mobile devices, including specialized agent models such as UI-TARS [21], MobileAgent [26], CogAgent [10], and etc. [16; 31; 28; 34; 9; 13; 27; 6; 35], as well as more general-purpose models like GPT-5 [19] and Gemini 2.5-pro [8].

Despite this progress, significant challenges remain when deploying these agent models in real-world environments. For example, existing agents often exhibit low task completion rates, slow response times, and limited ability to handle unexpected situations.

We first introduce the MobiMind-series models, which employ a multi-role architecture consisting of three distinct components: Planner, Decider, and Grounder. This design decouples task planning, reasoning, and execution, thereby facilitating seamless integration with various backend operation modes (e.g., GUI, XML, etc.). In addition, we propose an agent acceleration framework: AgentRR [7]. AgentRR records execution traces from the agent’s operation and abstracts them into multi-level experiences. A lightweight latent memory model determines whether the agent can leverage past experiences, thereby reducing the computational burden on VLMs/LLMs during task execution. With the AgentRR framework, the mobile agent can continuously enhance its efficiency and accuracy on recurring tasks.

To accurately evaluate the performance of our mobile agent models in real-world scenarios, we have surveyed existing mobile benchmark frameworks, including AITW [23], ANDROIDCONTROL [15], AndroidArena [29], AndroidWorld [22], GUIOdyssey [17], A3 [5], and etc. [14; 32; 11; 12; 20]. However, in practical usage scenarios, the presence of variable environmental factors, multiple correct execution trajectories, and the lack of deterministic verification mechanisms make it challenging for current benchmark frameworks to provide accurate assessments. To address these challenges, we design a novel mobile benchmarking framework: MobiFlow, based on Directed Acyclic Graphs (DAGs) and milestone events. MobiFlow defines multiple correct trajectories and employs a multi-level verification mechanism (e.g., text/icon matching, OCR, and LLM-based analysis) to enable precise and fine-grained evaluation of agent tasks. Furthermore, to mitigate the impact of environmental variability, MobiFlow enables replay-based evaluation by allowing different agents to execute pre-recorded static traces. This approach ensures a more controlled and deterministic assessment of agent performance.

Our experimental results demonstrate that, on real-world mobile scenarios (as evaluated by the MobiFlow Benchmark), the combination of MobiMind-Decider-7B and MobiMind-Grounder-3B outperforms general-purpose large language models (e.g., Gemini-2.5 Pro, GPT-5) as well as other specialized mobile agent models (such as UI-TARS-1.5-7B). Furthermore, MobiMind exhibits superior instruction-following capabilities, generates higher-quality reasoning, and achieves more reliable task termination.

2 Real-world Trajectory Collection

In this section, we present a framework for collecting real-world GUI task trajectories with rich contextual information and high data quality, while requiring minimal human efforts. The collected trajectories are further refined and used to train our agent models, enabling them to learn from diverse and realistic user interactions.

2.1 Agent Trajectory Collection

To ensure that the collected trajectories accurately reflect real-world user behavior, we developed a lightweight action recording tool for smartphones. During data collection, annotators interact with this tool, which captures and logs every user action before forwarding it to the device. The tool renders all bounding boxes (bbox) of interactive elements on the screen based on XML files that describe UI hierarchies. If a bbox is missing due to incomplete XML data, we employ OmniParser [18] to regenerate the corresponding bbox.

The recorded action space includes:

• •

Click(bbox: List[int]): Click at the center point of bbox using absolute coordinates.

• •

Input(text: str): Input text to a currently activated input area.

• •

Swipe(direction: Literal["UP","DOWN","LEFT","RIGHT"]): Swipe to direction.

• •

Done(): Signify the completion of the task.

For rare but important cases that are difficult to capture with the recording tool efficiently, such as closing a pop-up ad or waiting for it to disappear, we create a separate single-step dataset containing these cases and add an new wait action to the action space:

• •

Wait(sec: int): Wait for sec seconds before the next action.

For simpler tasks which a powerful pretrained VLM (e.g. Gemini-2.5) can handle, we directly prompt the VLM to execute these tasks and record the trajectories with a uniform format, which further improves the efficiency of data collection.

2.2 Agent Reasoning Reconstruction

The recorded trajectories only contain low-level information (e.g., raw coordinates or bounding boxes), lacking high-level semantics (e.g., task planning and environment observation) which is crucial for training intelligent and robust agent models. To address this issue, we propose a VLM-based reasoning reconstruction approach which leverages Gemini-2.5 to reconstruct the agent’s reasoning process based on original trajectories.

Specifically, we prompt the VLM to examine each action in the trajectory and provide a detailed reasoning, simulating an agent who executes the task following the ReAct [30] paradigm. Furthermore, for each concrete operation (e.g., click bbox), we also utilize the VLM to provide an action primitive (e.g., click search button) that abstractly describes this operation.

2.3 Post-collection Data Refinement

Initially collected trajectories may contain errors, have unbalanced distributions, or lack diversity. Besides applying a series of rule-based automated checks and manual reviews to remove erroneous or inconsistent samples, we further adopt the following data refinement strategies:

• •

Task Concatenation: We concatenate temporally dependent trajectories to form new trajectories corresponding to more complex tasks. In this way, the dataset diversity is boosted without extra efforts.

• •

Data Redistribution: Different trajectories tends to share similar prefixes while having diverse suffixes. When constructing training data, we perform sampling on the prefix actions while keeping all the suffix actions in the dataset, ensuring a more balanced distribution.

• •

History Augmentation: Besides samples with complete historical information, we also include samples with only partial history in training data. This approach avoids the model from overfitting to specific history patterns.

• •

Prompt Generalization: In real-world scenarios, agent’s input task descriptions varies across different users. However, each trajectory obtained during the initial data collection has only one corresponding task description, which can lead to weaker generalization abilities of the agent. Therefore, we assign more semantically equivalent task desciptions to every trajectory to build a more comprehensive training dataset.

• •

Corner-case Enhancement: With the single-step dataset obtained in Secion 2.1, we enhance the agent’s ability to handle more complicated corner cases (e.g., closing pop-up ads, terminating current task or returning to the previous page upon execution errors) by mixing single step desicion with history-based decision at training time. Experiments show that at test time, decider has stable capability of handling these problems at any time step without being interfered by the action history.

3 Training

We post-train decider and grounder on the basis of Qwen2.5-VL-7B and Qwen2.5-VL-3B [2], respectively. The post-training process incorporates 2 phases: a warm-up SFT phase and a two-stage curriculum GRPO [24] phase.

3.1 Warm-up SFT

In SFT phase, we use a subset of the training data with the following prompt templates to train decider and grounder:

Decider:

<image>

You are a phone-use AI agent. Now your task is "{task}".

Your action history is:

{history}

Please provide the next action based on the screenshot and your action

history. You should do careful reasoning before providing the action.

Your action space includes:

• Name: click, Parameters: target_element(a high-level description of the UI element to click).

• Name: swipe, Parameters: direction(one of UP, DOWN, LEFT, RIGHT).

• Name: input, Parameters: text(the text to input).

• Name: wait, Parameters: (no parameters, will wait for 1 second).

• Name: done, Parameters: (no parameters).

Your output should be a JSON object with the following format:

{"reasoning": "Your reasoning here", "action": "The next action (one of click, input, swipe, wait, done)", "parameters": {"param": "value"}}

Grounder:

<image>

Based on the screenshot, user’s intent and the description of the target UI element, provide the element’s bounding box using absolute coordinates.

User’s intent: {reasoning}

Target element’s description: {target_element}

Your output should be a JSON object with the following format:

{"bbox": [x1, y1, x2, y2]}

This warm-up phase empowers both models with basic format following and GUI agent capabilities, avoiding the reward signals in the subsequent RL phase being overly sparse which causes harm to training effectiveness.

3.2 Two-stage Curriculum GRPO with Self-evolution

Since MobiAgent adopts a multi-agent architecture, we simplify the multi-agent reinforcement learning process by incorporating 2 separate stages in GRPO phase to train the grounder and decider independently.

**Grounding GRPO. ** Firstly, we train the grounder with a rule-based reward function using the UI grounding dataset in Section 2. The reward function consists of two parts: an IoU reward $R_{\mathrm{iou}}$ and a center-point reward $R_{\mathrm{center}}$, reflecting the precision of the bounding box prediction of the grounder:

[TABLE]

where $b_{\mathrm{pred}},b_{gt}$ are the predicted and ground-truth bounding boxes, $\mathrm{IoU}\left(\cdot\right)$ is the Intersection over Union function, and $\alpha,\beta$ are hyperparameters.

**Grounder-as-RM GRPO. ** For click actions, since the decider’s output only contains a high-level description of the target element, it is unfeasible to evaluate the its quality by simply comparing it to the ground-truth. However, the grounder is able to map the high-level semantics to low-level and judgable actions. Therefore, after a high-precision grounder is obtained in the previous stage, we then let it serve as an outcome reward model (ORM) for training the decider.

Specifically, for each response generated in the rollout stage, if the action is a non-click operation, we apply a binary reward $R_{\mathrm{content}}$ by exact matching:

[TABLE]

where $a_{\mathrm{pred}},a_{\mathrm{gt}}$ are the predicted and ground-truth actions. For click action, the grounder predicts a bounding box of the target element in decider’s response, and directly returns $R=R_{\mathrm{center}}$ as the final reward.

**Curriculum Learning. **In Grounder-as-RM stage, we further adopt a curriculum learning [3]-based training approach to train the decider: Each action in the training dataset is labeled a binary difficulty with a series of predefined rules (e.g., action type frequency or manually assigned importance). In early training, each batch in model rollout only contains “easy” samples, and we gradually mix more “difficult” samples in the batches as the training progresses. This approach further addresses sparse rewards, accelerates model convergence and improves training robustness.

**Self-evolution. **To avoid unexpected agent behavior at test time due to error accumulation in the agents’ input context (i.e., the history part in decider’s input and the reasoning part in grounder’s input), we align the distribution of agent input at training time and test time via training-time-test. Specifically, after each round of training, we test the two models with new tasks and collect the test-time traces. Then we examine and correct the unsuccessful traces, and merge the corrected traces with successful traces to become a part of the training data of the next round, enabling the agent to self-evolve in an iterative way.

4 AgentRR: Record-Replay-based Agent Acceleration with Multi-level Experiences

In this section, we introduce a general-purpose agent acceleration framework: AgentRR, as well as our specific implementation in mobile scenario. AgentRR seamlessly supports any mobile agent models and applications without the need for additional development for specific tasks. Empirical evaluations show that ActTree enables GUI agents to achieve execution efficiency comparable to that of API agents, while preserving the generalization capability of GUI agents.

4.1 Multi-level Experiences

As shown in Figure 2, MobiAgent adopts a multi-agent architectural design, including a Planner (4B), a Decider (7B), and an Grounder (3B). The execution of a complete task requires the collaborative operation of these multiple models/agents. We record the output of each model as an experience, such that the outputs from different models form a multi-level experience hierarchy. The Grounder outputs concrete operation actions, such as the bounding box of the clicked object, representing the lowest-level experiences. The Decider outputs action primitives described in natural language. The Planner outputs the task plan, corresponding to high-level experiences. Lower-level experiences can be executed with higher efficiency, but typically offer weaker generalization, while higher-level experiences exhibit stronger generalization capabilities but are less efficient to execute (as they still require model inference).

To address this challenge, we propose a record-and-replay system specifically designed for agent frameworks. The core idea behind it is to efficiently determine which actions can be directly reused, without compromising the agent’s generalization capabilities. This mechanism is analogous to human latent memory, enabling swift execution of familiar actions while preserving the flexibility required for new situations.

4.2 Prefix Reusablity of GUI tasks

We observe the prefix reusablity of GUI tasks, which relies on the following insights:

• •

I1: The execution trajectories of similar tasks share identical prefixes.

• •

I2: Since the agent’s planning can be directly derived from task description, the length of shared trajectories (i.e., prefix actions) between two tasks can be predicted based on the similarity of their task descriptions.

According to I1, task trajectories can be cached using a tree structure, where similar tasks share the same path from the root node to their common ancestor node. According to I2, the agent can determine whether to reuse actions from historical tasks at the current step through simple and efficient semantic matching, thereby avoiding redundant model inference. By exploiting prefix reusability, we can efficiently records execution trajectories and attempts to replay cached actions for every execution step, thereby reducing the end-to-end task execution latency.

4.3 ActTree Structure

ActTree is a concrete implementation mechanism that accelerates mobile tasks by leveraging the principles of AgentRR. As shown in Figure 3 ActTree maintains a tree-structured representation analogous to UI Transition Graphs. Each node $v\in\mathcal{V}$ in the tree corresponds to a unique UI state, denoted as $\mathcal{UI}_{u}$, and maintains a set of outgoing edges $\mathcal{E}_{u}=\{e_{1},...,e_{n}\}$. Each edge $e=(u,v)\in\mathcal{E}$ represents a state transition, which is characterized by: (1) an action $\mathcal{A}_{e}$ that triggers this transition; and (2) a task list $\mathcal{T}_{e}=\{t_{1},...,t_{k}\}$, which records all historical tasks that executed action $\mathcal{A}_{e}$ to transition from $\mathcal{UI}_{u}$ to $\mathcal{UI}_{v}$.

The construction of the ActTree proceeds incrementally during the task executions. When a new action is performed, we first create a new node $v$ to represent the resulting UI state: $\mathcal{UI}_{v}$, along with a corresponding edge $e_{\mathrm{new}}=(u,v)$ annotated with the executed action $\mathcal{A}_{e_{\mathrm{new}}}$ and the current task description $t_{\mathrm{new}}$. To maintain the compactness of the ActTree, the system compacts the set of outgoing edges $\mathcal{E}_{u}=\{e_{1},\ldots,e_{n}\}$ from node $u$. For example, if there exists an edge $e_{\mathrm{old}}\in\mathcal{E}_{u}$ whose action $\mathcal{A}_{e_{\mathrm{old}}}$ matches the new action $\mathcal{A}_{e_{\mathrm{new}}}$, the system merges these two edges. The merge operation combines the task lists of the two edges by updating $\mathcal{T}_{e_{\mathrm{merge}}}\leftarrow\mathcal{T}_{e_{\mathrm{old}}}\cup\{t_{\mathrm{new}}\}$, while leaving the remaining edge structure unchanged.

To support more complex agent scenarios, such as recursive task invocation and collaboration among multiple applications, we further extend the ActTree structure to allow certain backtracking edges. For example, edges from child nodes back to parent nodes enable the agent to return to the parent’s UI state when an erroneous action is executed or when recursive tasks are invoked. Additionally, we also introduce cross-subtree edges with the additional transferred message to facilitate coordination across multiple tasks.

In addition to ActTree itself, we incorporate a Tracer component that is bound to ActTree to monitor task execution. During the execution of a new task, Tracer advances to a new node along with each UI transition. At each step, if Tracer identifies an existing outgoing edge $e$ whose action $\mathcal{A}_{e}$ can be reused for the current task (i.e., a cache hit), it replays $\mathcal{A}_{e}$ without invoking the agent model. Otherwise, the Tracer invokes the agent model to generate the next action.

4.4 Latent Memory Models

When using AgentRR framework to accelerate agent execution, a key challenge is determining how many prefix actions can be reused. To address this, we propose the “Latent Memory” model, which efficiently computes the similarity between two tasks and thereby determines the extent to which prefix actions can be reused. Specifically, the "Latent Memory" model is implemented in the form of the task embedding and task reranking models. These models typically have significantly fewer parameters than the planner or decider, resulting in lower response latency.

4.4.1 Task Embedding

The task embedding model is responsible for computing the similarity between two tasks, which is measured as the cosine similarity between their respective embedding vectors. Below, we provide the formal representation of the embedding at a specific layer of the ActTree for a given task.

[TABLE]

Where:

• •

$T_{i}$ represents the $i$-th task.

• •

$f$ is an instruction-aware task embedding model, and $f(T_{i},l)$ maps $T_{i}$ to a vector for the $l$-th layer of the ActTree. $l$ is in the instruction part of $f$’s input.

• •

$v_{i}^{(l)}$ is the embedding vector corresponding to the task $T_{i}$ and the layer $l$.

• •

$d$ is the dimension of the embedding vector, and $\mathbb{R}^{d}$ represents the $d$-dimensional real-valued vector space where the embedding vector resides.

To determine whether the action prefix of task $T_{i}$ can be reused by task $T_{j}$, we compare the cosine similarity of task embedding $S(v_{i}^{(l)},v_{j}^{(l)})$ with a predefined similarity threshold $\tau_{1}$. The stage-1 decision of whether $T_{i}$ can reuse the actions of $T_{j}$ at the $l$-th layer in ActTree is formally expressed as:

[TABLE]

4.4.2 Task Reranking

Another approach to implementing the latent model is to leverage task reranking. Given two candidate tasks, the latent model determines whether the action at a specific layer of ActTree can be reused, and directly return a score (the logit of "yes" token) between [math] and $1$. The task reranking mechanism serves as a complementary method to task embedding. In our implementation, we first utilize the task embedding model to compute similarity scores between the current task and all historical tasks for the current layer, then select a set of candidate tasks $\mathcal{C}$ with the highest similarity scores:

[TABLE]

where $T_{c}$ is the current task, $\mathcal{T}_{\text{hist}}$ represents all historical tasks recorded in outgoing edges of the current ActTree node.

Subsequently, we perform task reranking by comparing each candidate task with the current task. If there exists at least one candidate whose reranking score with the current task is greater than another predefined confidence threshold $\tau_{2}$, the action at the corresponding node in ActTree is considered reusable for the current task. This process enables the latent model to make more accurate reuse decisions compared to relying solely on the embedding model. The stage-2 (final) reuse decision is expressed as:

[TABLE]

Where:

• •

$g$ is an instruction-aware task reranking model, and $g(T_{i},T_{j},l)$ returns a confidence score of the action reuseability between $T_{i}$ and $T_{j}$ at $l$-th layer.

4.4.3 Latent Memory Model Training

We select Qwen-Embedding and Qwen-Reranker [33] as our base models. All training data are collected through our agent model by recording actual action traces on mobile devices, ensuring behavioral consistency between the Latent Memory Model and the Agent Model.

For the task embedding model, for a given ActTree layer $l$ and a task $T$, other tasks that share an $l$-step action prefix with $T$ are considered positive samples, while tasks sharing fewer than $l$ steps are considered negative samples. Formally, for $v=f(T_{i},l)$, the positive and negative samples are sampled in the following way:

[TABLE]

where $\mathrm{pre}(\cdot,\cdot)$ gives the length of the shared action prefix of two tasks, and we calculate the InfoNCE [25] loss with a temperature parameter $\tau$:

[TABLE]

For the task reranking model, the determination of positive and negative samples is consistent with the embedding model. The positive samples are labeled as “yes” and negative samples as “no”, and we use SFT loss to train the model.

4.5 Latent Memory Invalidation and Eviction

Experiences may become obsolete as the environment changes, such as application updates or device replacements. Compared to high-level experiences, low-level experiences (e.g., action coordinates output by the Actor) tend to become outdated more frequently. In the AgentRR system, it is essential to promptly detect and update obsolete experiences to ensure the correctness of subsequent action reuse.

4.5.1 UI Change Detection

Numerous modern applications utilize dynamic UI components (e.g., the items in search results page change according to user preferences). UI change can lead to incorrect action replay in ActTree when identical actions could yield different results. ActTree leverages OmniParser, a screen parsing tool based on small vision models, to detect and mitigate this anomaly. When executing an action with a target UI element (e.g., a click action) for the first time, Tracer invokes OmniParser to extract the target element’s bounding box and content. Prior to replaying an action during another task’s execution, Tracer invokes OmniParser again to verify whether the UI content within the previous bounding box has changed. If a UI change is detected, the cache hit is discarded and the cached action is invalidated.

4.5.2 False Positive of Reuse Decision

If the latent model outputs a reuse decision of true, but the action cannot actually be reused for the current task, this results in a false positive decision. However, false positives in the AgentRR system do not lead to final execution errors. After a false positive action is performed, there will be a significant discrepancy between the current UI state and the previously recorded UI state, causing a failure of UI checking. Subsequently, AgentRR will invoke the Decider module again to determine the appropriate action for the current interface, which is typically a back operation.

4.5.3 LRU Experience Eviction

In addition to experience obsolete, some experiences may rarely be used, resulting in unnecessary consumption of storage resources. To address this, ActTree implements a Least Recently Used (LRU) eviction policy to prevent out-of-memory issues caused by unused experiences. When the cache reaches its maximum size, the least recently used experience is removed to make space for new experience.

4.6 Speculative Replay via Shortcuts

Task trajectories usually consist of a series of sub-tasks. A sub-task is defined as an action sequence following a fixed pattern, e.g., the “search” sub-task includes 3 steps: clicking the search bar to activate it, entering the query, and clicking the search button to execute the search. As in the aforementioned example, different trajectories of a sub-task can share both prefixes and suffixes (both step 1 and step 3 are reusable). We refer to frequently recurring sub-tasks as shortcuts. By identifying shortcuts in the tree, ActTree enables speculative replay.

A shortcut is defined as a set of sub-paths in the ActTree, denoted as $S$. For any sub-path pair $(p_{i},p_{j})\in S^{2}$, $p_{i}$ and $p_{j}$ satisfy the following constraints:

• •

They start from the same ActTree node and have identical lengths.

• •

Their action type sequences before the last action are identical.

• •

Their last actions are identical, denoted as $\mathcal{A}_{S}^{*}$.

When Tracer reaches a newly created node with no outgoing edges, there are no available experiences to replay. In this scenario, Tracer examines whether the current partially completed task $T$ satisfies the following condition: for some $S$, if $A_{S}^{*}$ is the next action for $T$, a sub-path of $T$ can be joined with $S$. If this condition is met, Tracer speculatively selects $A_{S}^{*}$ as a candidate for the next action. In this manner, ActTree expands the scope of experience utilization and enables the discovery of additional replay opportunities.

5 MobiFlow: a DAG-Based Benchmark Framework for Mobile Agents

Current mobile benchmarks primarily focus on system applications and tasks with clearly defined evaluation criteria. However, in real-world mobile agent scenarios, operations often involve third-party applications and lack explicit verification conditions. As a result, existing mobile agent benchmarks fail to accurately reflect the model’s capabilities in mobile-side scenarios. To address this limitation, we propose an evaluation framework tailored to the manipulation of third-party mobile applications. By leveraging a Directed Acyclic Graph (DAG) to model the complex dependencies and sequential constraints inherent in mobile application tasks, our framework enables a more realistic assessment of model performance in practical on-device agent scenarios.

5.1 Framework Architecture

Figure 4 illustrates our benchmark framework. Given the complexity and variability of real-world mobile applications, we define multiple validation methods, diverse task trajectory specifications, and both online and offline trajectory collections. These design choices aim to accurately capture the agent’s capabilities in real-world scenarios. The overall architecture of the benchmarking framework is as follows:

Trace Collection: We categorize agent execution traces into two types: online and offline. Online traces are collected through real interactions between the agent and mobile devices, logging all agent actions as well as UI screenshots from various pages. In contrast, offline traces are obtained by predefining the agent’s task execution process and recording information such as action primitives and XML data generated. Additionally, we merge shared steps across different trajectories for the same task. By leveraging offline traces, we can conduct stable evaluations of the agent’s performance, effectively eliminating the impact of dynamic environmental variables. 2. 2.

DAG-based Task Definition: Each task is defined by a configuration file that specifies the structure of a DAG. In this graph, nodes represent key milestones in task execution, such as UI states or critical operations, while edges capture the dependencies and action transitions between these milestones. 3. 3.

Validation Engine: The engine parses both the task’s DAG configuration and the agent’s execution trace. It traverses the DAG alongside the trace, searching within the actual execution trace for milestone events and their dependencies as defined in the DAG. 4. 4.

Result Generation:Based on this traversal, the engine determines whether a fully validated path or a partially validated path exists. Tasks that successfully complete all milestone events defined in the DAG are awarded the full score, while tasks that accomplish only a subset of milestone events may receive partial credit, as specified by user-defined criteria.

In real-world scenarios, application responses are highly dependent on factors such as application version, environment, and interaction patterns. Moreover, there is often a lack of systematic interfaces for quickly verifying whether milestone events have been completed. To address these challenges, we design a multi-level verification mechanism to ensure the accuracy of evaluating agent tasks on the mobile side.

5.2 Core Validation Capabilities

The framework supports a hierarchical approach to condition checking for robust and resilient validation.

Multi-level Condition Checkers: Each node in the DAG can be associated with one or more conditions that must be met. The supported checker types include:

• •

Text Matching: Verifies the presence of specific strings or patterns in the UI.

• •

Regular Expression Matching: Allows for more complex text-based validation using regular expressions.

• •

UI State Analysis: Examines the underlying view hierarchy (e.g., via XML dumps) to confirm the state of UI elements.

• •

Icon Detection: Utilizes computer vision techniques to identify the presence of specific icons.

• •

OCR Process: If neither XML files nor the page text is available, we employ OCR techniques to extract and verify text directly from screenshots.

• •

MLLM-as-a-Judge: Employs a multi-modal LLM to make a holistic judgment about the UI state based on the current key frame and contextual information.

Progressive Escalation Strategy: The checker starts with the lightest, fastest verification path. When text matching is required, it first parses the XML file and instantly confirms the current page state. Only if the XML source is absent or a plain-text checker is disabled does the system escalate to heavier techniques, like OCR or LLM-based reasoning, to decide whether the milestone has been reached.

Multi-path DAG Validation: The DAG structure inherently supports tasks with multiple valid completion trajectories. For tasks that can be accomplished through different approaches, we specify whether the milestone nodes are dependent on others or can proceed independently in parallel. This allows the DAG representation of a task to include multiple branches and paths. To better express the relationships between different paths, we define the AND and OR conjunctions. The AND indicates that all prerequisite milestone events from the preceding nodes must be satisfied before proceeding. In contrast, the OR stipulates that satisfying any single milestone event within the set is sufficient to trigger validation of the subsequent node. For a given task, if multiple paths can lead to its completion, the task is considered accomplished as long as any one of these paths is successfully validated.

To ensure the correct matching between milestone event nodes defined in the DAG and nodes in the trace, we introduce a synchronized backtracking mechanism for DAG and trace nodes. Specifically, when the checker detects that a certain path in the DAG cannot be validated, it backtracks to the previous branching milestone event and selects an alternative path for verification. Simultaneously, the checker also backtracks within the actual trace until it reaches the milestone node corresponding to the branching point.

Dynamic Condition Matching: MobiFlow supports task definition via templates and dynamically extracts conditions from the task description during execution. For example, if the task is “add milk to the shopping cart” the validator can automatically identify “milk” as the key object and subsequently generate corresponding milestone event nodes in sequence.

Icon Recognition: MobiFlow leverages OpenCV [4] for icon recognition and supports multi-scale matching. By controlling the similarity threshold, it can robustly identify UI elements, even in the presence of minor variations in size or appearance.

5.3 Construct the MobiFlow Benchmark for Real-world Mobile Agent

To construct a benchmark that evaluates agent capabilities in realistic mobile environments, we select a range of representative mobile applications and scenarios, encompassing domains such as social networking, music and video, shopping, travel, food delivery, etc. Each test case defines milestone events using a template-based approach, ensuring sufficient generalizability of test cases. Additionally, for each application, we design task cases with varying levels of difficulty, ranging from simple to complex, to accurately assess the capabilities of agent models.

In our actual testing process, we meticulously select tasks that can be correctly executed in the current environment, thereby minimizing test failures caused by task misconfigurations rather than agent performance. For test cases that result in abnormal or failed executions, we further employ manual verification to ensure the accuracy and reliability of the evaluation.

Due to the iteration of applications and the variability of operating environments, the absolute scores obtained from MobiFlow are primarily for reference. However, the relative scores among different agent models provide a more accurate reflection of their capabilities in mobile agent scenarios. Additionally, we can record the correct action sequences from a given benchmark run as offline traces, including multiple valid trajectories for the same task, and enable different models to replay these offline traces for evaluation. This approach eliminates the influence of dynamic environmental factors, thereby making the evaluation results more deterministic.

Nevertheless, evaluating with offline traces may lead to false negatives. For example, an agent model could discover a new successful trajectory; however, such a correct outcome would be erroneously classified as a failure under offline-trace-based evaluation.

6 Evaluation

6.1 Task Completion Score

We evaluated the task completion rates of various models in real-world mobile scenarios. Our test suite covers the current widely-used mobile applications in China, as well as common tasks, such as travel, shopping, entertainment and social networking. For each application, we defined a set of test case templates with multiple difficulty levels, where each template consists of milestone events structured in the form of a DAG. Prior to formal testing, we generated three real-world test cases for each type of template. All test cases are defined within our MobiFlow benchmark framework.

Figure 5 presents a comparison of task completion rates between MobiAgent (MobiMind-Decider-7B + MobiMind-Grounder-3B) and several other VLM models, including the general-purpose LLMs (GPT-5, Gemini-2.5-pro) and the state-of-the-art GUI Agent model UI-TARS-1.5-7B. To ensure the accuracy of our evaluation, we manually verified all failed tasks. For cases caused by app anomalies, network interruptions, or non-existent tasks, we conducted repeated testing. In addition, we imposed a termination penalty on tasks where the agent could not complete or exit properly.

MobiAgent achieves the highest task completion rates across the majority of applications. In complex tasks such as shopping and food delivery, MobiAgent demonstrates superior performance in task understanding and decomposition, instruction following, and handling of exceptional cases. During execution, GPT and Gemini tend to input object descriptions, types, and details such as departure time and locations, directly into the search box. Because some apps support AI-powered search capabilities, these models can bypass most complex interactive steps, resulting in relatively higher task completion rates. However, such coarse-grained operation strategies lead to a significant drop in completion rates when faced with applications that do not support AI search.

Furthermore, we observed that existing agent models still suffer from task non-termination issues, such as the infinite repetition of certain actions. Specifically, GPT failed to properly terminate tasks in 11 application categories, while Gemini exhibited similar problems in 3 categories. In contrast, MobiAgent consistently achieved correct task termination across all evaluated scenarios.

6.2 Accelerating the Mobile Agent using AgentRR

In addition to evaluating the task completion rates of agents, we also measure the speedup achieved by the AgentRR acceleration framework across various scenarios. By leveraging ActTree and a latent memory model, AgentRR can rapidly reuse historical agent trajectories, thereby reducing the inference overhead of VLMs. Consequently, the acceleration ratio of AgentRR in real-world mobile scenarios is positively correlated with the proportion of nodes in ActTree that can be reused.

To simulate different user behaviors on mobile devices, we constructed two types of user task distributions: a uniform distribution and a power law distribution. In the uniform distribution, users evenly utilize various application functionalities and randomly choose task parameters (e.g., search objects). In the power law distribution, 80% of user requests are concentrated within 20% of tasks, which more closely aligns with real-world user habits.

As Figure 6 shows, under the uniform distribution, AgentRR achieves an action replay rate of 30%–60%. Under the power law distribution, the action replay rate increases to 60%–85%. Moreover, owing to the accuracy of the latent model, the correctness of action replay exceeds 99% in our test cases.

In the real-world deployment scenarios, the performance advantage of AgentRR becomes even more pronounced, as the latent memory model can be deployed directly on edge devices, thereby reducing network communication overhead. In complex task scenarios such as food delivery and online shopping, AgentRR achieves an average performance improvement of 2 to 3 times compared to baseline approaches.

7 Conclusion

In this paper, we address the challenges of building, evaluating and deploying mobile agent by introducing a comprehensive, full-stack solution encompassing the MobiMind model series, the AgentRR acceleration framework, and the MobiFlow evaluation benchmark. Experiments show that MobiAgent achieves state-of-the-art task completion rates that surpass both general-purpose models like GPT-5 and Gemini-2.5 Pro, and other specialized mobile agents in the MobiFlow benchmark. Moreover, The experience replay mechanism of AgentRR framework delivers a 2-3x optimization on agent’s task completion latency in real-world scenarios. These results demonstrate the effectiveness of our approach in creating customizable and practical mobile agents.