Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skills

TL;DR

Mirage-1 introduces a hierarchical skills framework and a skill-augmented search algorithm to enhance multimodal GUI agents, significantly improving their performance in real-world long-horizon tasks across multiple platforms.

Contribution

The paper proposes a hierarchical multimodal skills module and a skill-augmented Monte Carlo Tree Search algorithm, enabling GUI agents to better generalize knowledge and perform long-horizon tasks online.

Findings

Mirage-1 outperforms previous agents by up to 79% on new benchmarks.

Hierarchical skills improve long-horizon task planning.

SA-MCTS reduces action search space effectively.

Abstract

Recent efforts to leverage the Multi-modal Large Language Model (MLLM) as GUI agents have yielded promising outcomes. However, these agents still struggle with long-horizon tasks in online environments, primarily due to insufficient knowledge and the inherent gap between offline and online domains. In this paper, inspired by how humans generalize knowledge in open-ended environments, we propose a Hierarchical Multimodal Skills (HMS) module to tackle the issue of insufficient knowledge. It progressively abstracts trajectories into execution skills, core skills, and ultimately meta-skills, providing a hierarchical knowledge structure for long-horizon task planning. To bridge the domain gap, we propose the Skill-Augmented Monte Carlo Tree Search (SA-MCTS) algorithm, which efficiently leverages skills acquired in offline environments to reduce the action search space during online tree exploration. Building on HMS, we propose Mirage-1, a multimodal, cross-platform, plug-and-play GUI agent. To validate the performance of Mirage-1 in real-world long-horizon scenarios, we constructed a new benchmark, AndroidLH. Experimental results show that Mirage-1 outperforms previous agents by 32\%, 19\%, 15\%, and 79\% on AndroidWorld, MobileMiniWob++, Mind2Web-Live, and AndroidLH, respectively. Project page: https://cybertronagent.github.io/Mirage-1.github.io/