UI-Mem: Self-Evolving Experience Memory for Online Reinforcement Learning in Mobile GUI Agents

TL;DR

UI-Mem introduces a hierarchical experience memory framework for online reinforcement learning in GUI agents, enabling effective experience transfer and continuous self-evolution, leading to improved performance and generalization across applications.

Contribution

The paper presents a novel hierarchical experience memory and stratified sampling method for online RL in GUI agents, facilitating cross-task transfer and continuous self-improvement.

Findings

UI-Mem outperforms traditional RL baselines on GUI benchmarks.

It demonstrates strong generalization to unseen applications.

The framework effectively transfers knowledge across tasks and applications.

Abstract

Online Reinforcement Learning (RL) offers a promising paradigm for enhancing GUI agents through direct environment interaction. However, its effectiveness is severely hindered by inefficient credit assignment in long-horizon tasks and repetitive errors across tasks due to the lack of experience transfer. To address these challenges, we propose UI-Mem, a novel framework that enhances GUI online RL with a Hierarchical Experience Memory. Unlike traditional replay buffers, our memory accumulates structured knowledge, including high-level workflows, subtask skills, and failure patterns. These experiences are stored as parameterized templates that enable cross-task and cross-application transfer. To effectively integrate memory guidance into online RL, we introduce Stratified Group Sampling, which injects varying levels of guidance across trajectories within each rollout group to maintain outcome diversity, driving the unguided policy toward internalizing guided behaviors. Furthermore, a Self-Evolving Loop continuously abstracts novel strategies and errors to keep the memory aligned with the agent's evolving policy. Experiments on online GUI benchmarks demonstrate that UI-Mem significantly outperforms traditional RL baselines and static reuse strategies, with strong generalization to unseen applications. Project page: https://ui-mem.github.io