When Continual Learning Moves to Memory: A Study of Experience Reuse in LLM Agents

TL;DR

This paper investigates how external memory in LLM agents affects continual learning, revealing that memory management, not just model updates, is crucial for overcoming learning challenges.

Contribution

The study introduces a (k,v) framework to analyze memory design in LLM agents, highlighting the impact of memory representation and organization on continual learning.

Findings

Abstract procedural memories transfer more reliably than detailed trajectories.

Negative transfer harms hard cases more than easy ones.

Memory organization can cause forgetting despite enabling forward transfer.

Abstract

Memory-augmented LLM agents offer an appealing shortcut to continual learning: rather than updating model parameters, they accumulate experience in external memory, seemingly sidestepping the stability-plasticity dilemma of parametric learning. We show that this challenge does not disappear but resurfaces at the memory level. Under a limited context window, old and new experiences compete during retrieval, relocating the continual-learning bottleneck from parameter updates to memory access. To study this phenomenon, we introduce a (k,v) framework that disentangles two fundamental design axes of external memory: how experience is represented and how it is organized for retrieval. Across sequential-task experiments in ALFWorld and BabyAI, we find that abstract procedural memories transfer more reliably than detailed trajectories, while negative transfer disproportionately harms the hard cases. Moreover, finer-grained memory organization is not universally beneficial: designs that yield strong forward transfer can simultaneously induce severe forgetting. Together, these results reveal that external memory does not resolve the continual-learning problem; it reshapes it into a problem of memory representation and retrieval design.