MobiFlow: Real-World Mobile Agent Benchmarking through Trajectory Fusion

TL;DR

MobiFlow is a new benchmarking framework for mobile agents that better reflects real-world scenarios by using multi-trajectory fusion to evaluate third-party app interactions.

Contribution

It introduces MobiFlow, an evaluation framework that aligns more closely with real-world mobile agent tasks by supporting dynamic interactions and compressing state space.

Findings

MobiFlow covers 20 third-party apps with 240 real-world tasks.

It achieves higher alignment with human assessments compared to AndroidWorld.

Supports dynamic interaction and efficient state space compression.

Abstract

Mobile agents can autonomously complete user-assigned tasks through GUI interactions. However, existing mainstream evaluation benchmarks, such as AndroidWorld, operate by connecting to a system-level Android emulator and provide evaluation signals based on the state of system resources. In real-world mobile-agent scenarios, however, many third-party applications do not expose system-level APIs to determine whether a task has succeeded, leading to a mismatch between benchmarks and real-world usage and making it difficult to evaluate model performance accurately. To address these issues, we propose MobiFlow, an evaluation framework built on tasks drawn from arbitrary third-party applications. Using an efficient graph-construction algorithm based on multi-trajectory fusion, MobiFlow can effectively compress the state space, support dynamic interaction, and better align with real-world third-party application scenarios. MobiFlow covers 20 widely used third-party applications and comprises 240 diverse real-world tasks, with enriched evaluation metrics. Compared with AndroidWorld, MobiFlow's evaluation results show higher alignment with human assessments and can guide the training of future GUI-based models under real workloads.