Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigation

TL;DR

This paper introduces Chain-of-Memory (CoM), a method for explicitly modeling short-term and long-term memory in GUI agents, improving their understanding and performance in complex cross-application tasks.

Contribution

The paper proposes CoM, a novel explicit memory mechanism for GUI agents, and introduces GUI Odyssey-CoM, a large dataset for training and evaluating memory-enhanced GUI agents.

Findings

CoM significantly improves GUI agents' performance in cross-application tasks.

GUI Odyssey-CoM enables smaller models to match larger models' memory capabilities.

Experimental results show enhanced task understanding with explicit memory modeling.

Abstract

Multimodal large language models (MLLMs) are attracting growing attention in the development of Graphical User Interface (GUI) agents. Existing approaches often rely on historical screenshots or actions to implicitly represent the task state. This reliance poses challenges for GUI agents in accurately understanding task states and underscores the absence of effective mechanisms to store critical information in complex and lengthy cross-app tasks. To address these challenges, we propose Chain-of-Memory (CoM), a novel approach for explicitly modeling short-term and long-term memory in GUI agents. CoM achieves this by capturing action descriptions, integrating task-relevant screen information, and maintaining a dedicated memory module to store and manage this information. By leveraging explicit memory representations, CoM enables GUI agents to better understand task states and retain critical historical information persistently. To equip GUI agents with memory management capabilities and evaluate the effectiveness of CoM, we developed the GUI Odyssey-CoM, a dataset comprising 111k screen-action pairs annotated with Chain-of-Memory. Experimental results demonstrate that CoM significantly improves GUI agents' performance in cross-application tasks. Additionally, GUI Odyssey-CoM enables 7B models to achieve memory management capabilities comparable to 72B models. The dataset and code will be open-sourced.