MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devices

TL;DR

MobileExperts introduces a multi-agent, tool-based approach for mobile devices that dynamically assembles expert teams to handle complex tasks efficiently, reducing reasoning costs and improving performance across various task complexities.

Contribution

This work pioneers the integration of tool formulation and multi-agent collaboration in mobile device automation, enhancing handling of complex tasks with reduced reasoning costs.

Findings

Outperforms existing methods across all intelligence levels.

Achieves approximately 22% reduction in reasoning costs.

Validates effectiveness on a new hierarchical intelligence benchmark.

Abstract

The attainment of autonomous operations in mobile computing devices has consistently been a goal of human pursuit. With the development of Large Language Models (LLMs) and Visual Language Models (VLMs), this aspiration is progressively turning into reality. While contemporary research has explored automation of simple tasks on mobile devices via VLMs, there remains significant room for improvement in handling complex tasks and reducing high reasoning costs. In this paper, we introduce MobileExperts, which for the first time introduces tool formulation and multi-agent collaboration to address the aforementioned challenges. More specifically, MobileExperts dynamically assembles teams based on the alignment of agent portraits with the human requirements. Following this, each agent embarks on an independent exploration phase, formulating its tools to evolve into an expert. Lastly, we develop a dual-layer planning mechanism to establish coordinate collaboration among experts. To validate our effectiveness, we design a new benchmark of hierarchical intelligence levels, offering insights into algorithm's capability to address tasks across a spectrum of complexity. Experimental results demonstrate that MobileExperts performs better on all intelligence levels and achieves ~ 22% reduction in reasoning costs, thus verifying the superiority of our design.