MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents

TL;DR

MementoGUI introduces a modular, learned memory control framework for long-horizon GUI agents, enhancing their ability to maintain and utilize task-relevant visual and textual history for improved decision-making.

Contribution

The paper presents MementoGUI, a plug-in agentic memory system with learned controllers for online memory management, enabling better long-term GUI task performance without finetuning the backbone.

Findings

MementoGUI improves GUI agent performance over baseline methods.

Larger MementoCore backbones lead to stronger memory-augmented control.

The framework effectively handles long-horizon decision-making in GUI tasks.

Abstract

Recent GUI agents have made substantial progress in visual grounding and action prediction, yet they remain brittle in long-horizon tasks that require maintaining task state across many interface transitions. Existing agents typically rely on raw history replay or text-only memory, which either overwhelms the model with redundant screenshots or discards localized visual evidence needed for future decisions. To address these limitations, we introduce \textbf{MementoGUI}, a plug-in agentic memory framework that equips MLLM-based GUI agents with \textbf{MementoCore}, a learned controller for online memory selection, compression, and retrieval. Rather than treating interaction history as a fixed context, MementoGUI formulates long-horizon GUI control as an online memory-control problem: working memory selectively preserves task-relevant interface events with textual summaries and ROI-level visual evidence, while episodic memory retrieves reusable past trajectories through learned relevance selection. MementoCore modularizes memory control into specialized operators for step processing, memory compression, episodic writing, and episodic selection, enabling plug-in memory augmentation without finetuning the GUI agent backbone. We further develop a scalable data curation pipeline that converts computer-use trajectories into memory-controller training data, introduce \textbf{MementoGUI-Bench} for evaluating long-horizon decision-making in GUI agents, and design MLLM-based metrics for semantic action matching, task progress, and memory consistency. Experiments on GUI-Odyssey, MM-Mind2Web, and MementoGUI-Bench show that MementoGUI consistently improves GUI agents over no-history, history-replay, and text-only memory baselines, with larger MementoCore backbones further strengthening memory-augmented GUI control.