Memory Retention Is Not Enough to Master Memory Tasks in Reinforcement Learning

TL;DR

This paper introduces a new benchmark to evaluate memory rewriting in reinforcement learning, revealing that classic recurrent models outperform modern structured and transformer-based memories in adaptive updating tasks.

Contribution

It presents a benchmark for continual memory updating in RL, compares different architectures, and highlights the need for memory mechanisms balancing retention and updating.

Findings

Recurrent models show greater flexibility in memory rewriting.

Structured memories succeed only under narrow conditions.

Transformers often fail beyond trivial retention cases.

Abstract

Effective decision-making in the real world depends on memory that is both stable and adaptive: environments change over time, and agents must retain relevant information over long horizons while also updating or overwriting outdated content when circumstances shift. Existing Reinforcement Learning (RL) benchmarks and memory-augmented agents focus primarily on retention, leaving the equally critical ability of memory rewriting largely unexplored. To address this gap, we introduce a benchmark that explicitly tests continual memory updating under partial observability, i.e. the natural setting where an agent must rely on memory rather than current observations, and use it to compare recurrent, transformer-based, and structured memory architectures. Our experiments reveal that classic recurrent models, despite their simplicity, demonstrate greater flexibility and robustness in memory rewriting tasks than modern structured memories, which succeed only under narrow conditions, and transformer-based agents, which often fail beyond trivial retention cases. These findings expose a fundamental limitation of current approaches and emphasize the necessity of memory mechanisms that balance stable retention with adaptive updating. Our work highlights this overlooked challenge, introduces benchmarks to evaluate it, and offers insights for designing future RL agents with explicit and trainable forgetting mechanisms. Code: https://quartz-admirer.github.io/Memory-Rewriting/