Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System

TL;DR

This paper introduces UI-NEXUS, a new benchmark for evaluating compositional generalization in mobile agents, and proposes AGENT-NEXUS, a scheduling system that improves task success rates by decomposing complex tasks into atomic subtasks.

Contribution

The paper presents UI-NEXUS, a comprehensive benchmark for compositional tasks, and introduces AGENT-NEXUS, a scheduling system that enhances existing mobile agents' performance on these tasks.

Findings

Existing agents struggle with compositional tasks, showing failure modes like under- and over-execution.

UI-NEXUS presents significant challenges, revealing the generalization gap.

AGENT-NEXUS improves task success rates by 24-40% on compositional tasks.

Abstract

Autonomous agents powered by multimodal large language models have been developed to facilitate task execution on mobile devices. However, prior work has predominantly focused on atomic tasks -- such as shot-chain execution tasks and single-screen grounding tasks -- while overlooking the generalization to compositional tasks, which are indispensable for real-world applications. This work introduces UI-NEXUS, a comprehensive benchmark designed to evaluate mobile agents on three categories of compositional operations: Simple Concatenation, Context Transition, and Deep Dive. UI-NEXUS supports interactive evaluation in 20 fully controllable local utility app environments, as well as 30 online Chinese and English service apps. It comprises 100 interactive task templates with an average optimal step count of 14.05. Experimental results across a range of mobile agents with agentic workflow or agent-as-a-model show that UI-NEXUS presents significant challenges. Specifically, existing agents generally struggle to balance performance and efficiency, exhibiting representative failure modes such as under-execution, over-execution, and attention drift, causing visible atomic-to-compositional generalization gap. Inspired by these findings, we propose AGENT-NEXUS, a lightweight and efficient scheduling system to tackle compositional mobile tasks. AGENT-NEXUS extrapolates the abilities of existing mobile agents by dynamically decomposing long-horizon tasks to a series of self-contained atomic subtasks. AGENT-NEXUS achieves 24% to 40% task success rate improvement for existing mobile agents on compositional operation tasks within the UI-NEXUS benchmark without significantly sacrificing inference overhead. The demo video, dataset, and code are available on the project page at https://ui-nexus.github.io.