How Mobile World Model Guides GUI Agents?

TL;DR

This paper investigates how different representations in mobile world models affect GUI agent performance, emphasizing multimodal data and the utility of generated trajectories for training and guidance.

Contribution

It introduces and evaluates four modalities of world models, demonstrating their impact on performance and providing insights into their use for training and online guidance.

Findings

Renderable code reconstruction offers high fidelity and effective supervision.

Text-based feedback is more robust for out-of-distribution execution.

Generated trajectories improve task performance but do not match original data distribution.

Abstract

Recent advances in vision-language models have enabled mobile GUI agents to perceive visual interfaces and execute user instructions, but reliable prediction of action consequences remains critical for long-horizon and high-risk interactions. Existing mobile world models provide either text-based or image-based future states, yet it remains unclear which representation is useful, whether generated rollouts can replace real environments, and how test-time guidance helps agents of different strengths. To answer the above questions, we filter and annotate mobile world-model data, then train world models across four modalities: delta text, full text, diffusion-based images, and renderable code. These models achieve SoTA performance on both MobileWorldBench and Code2WorldBench. Furthermore, by evaluating their downstream utility on AITZ, AndroidControl, and AndroidWorld, we obtain three findings. First, renderable code reconstruction achieves high in-distribution fidelity and provides effective multimodal supervision for data construction, while text-based feedback is more robust for online out-of-distribution (OOD) execution. Second, world-model-generated trajectories can provide transferable interaction experience in the training process and improve agents' end-to-end task performance, although these data do not preserve the original distribution. Last, for overconfident mobile agents with low action entropy, posterior self-reflection provides limited gains, suggesting that world models are more effective as prior perception or training supervision than as universal post-hoc verifiers.