Hybrid Self-evolving Structured Memory for GUI Agents

TL;DR

This paper introduces HyMEM, a graph-based, self-evolving structured memory system inspired by the brain, which significantly enhances GUI agents' ability to handle complex, long-horizon tasks with diverse interfaces.

Contribution

HyMEM is a novel hybrid memory architecture combining symbolic and continuous data, enabling multi-hop retrieval and self-updating, improving GUI agent performance over prior flat memory methods.

Findings

HyMEM improves GUI agent performance across multiple benchmarks.

HyMEM enables 7B/8B models to outperform some closed-source models.

HyMEM boosts Qwen2.5-VL-7B accuracy by +22.5%.

Abstract

The remarkable progress of vision-language models (VLMs) has enabled GUI agents to interact with computers in a human-like manner. Yet real-world computer-use tasks remain difficult due to long-horizon workflows, diverse interfaces, and frequent intermediate errors. Prior work equips agents with external memory built from large collections of trajectories, but relies on flat retrieval over discrete summaries or continuous embeddings, falling short of the structured organization and self-evolving characteristics of human memory. Inspired by the brain, we propose Hybrid Self-evolving Structured Memory (HyMEM), a graph-based memory that couples discrete high-level symbolic nodes with continuous trajectory embeddings. HyMEM maintains a graph structure to support multi-hop retrieval, self-evolution via node update operations, and on-the-fly working-memory refreshing during inference. Extensive experiments show that HyMEM consistently improves open-source GUI agents, enabling 7B/8B backbones to match or surpass strong closed-source models; notably, it boosts Qwen2.5-VL-7B by +22.5% and outperforms Gemini2.5-Pro-Vision and GPT-4o.