AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generation

TL;DR

AutoDroid-V2 leverages small language models and code generation techniques to improve mobile UI task automation on-device, enhancing privacy, reducing latency, and lowering resource consumption compared to existing large-model-based agents.

Contribution

It introduces a document-centered approach for generating UI automation code with small language models, enabling efficient on-device execution and improved success rates.

Findings

Higher success rates in task automation

Lower latency and token consumption

Effective on-device execution of UI scripts

Abstract

Large language models (LLMs) have brought exciting new advances to mobile UI agents, a long-standing research field that aims to complete arbitrary natural language tasks through mobile UI interactions. However, existing UI agents usually demand powerful large language models that are difficult to be deployed locally on end-users' devices, raising huge concerns about user privacy and centralized serving cost. Inspired by the remarkable coding abilities of recent small language models (SLMs), we propose to convert the UI task automation problem to a code generation problem, which can be effectively solved by an on-device SLM and efficiently executed with an on-device code interpreter. Unlike normal coding tasks that can be extensively pre-trained with public datasets, generating UI automation code is challenging due to the diversity, complexity, and variability of target apps. Therefore, we adopt a document-centered approach that automatically builds fine-grained API documentation for each app and generates diverse task samples based on this documentation. By guiding the agent with the synthetic documents and task samples, it learns to generate precise and efficient scripts to complete unseen tasks. Based on detailed comparisons with state-of-the-art mobile UI agents, our approach effectively improves the mobile task automation with significantly higher success rates and lower latency/token consumption. Code is open-sourced at https://github.com/MobileLLM/AutoDroid-V2.