Is One Score Enough? Rethinking the Evaluation of Sequentially Evolving LLM Memory

TL;DR

This paper introduces SeqMem-Eval, a diagnostic framework for evaluating LLM memory in sequential tasks, revealing that traditional metrics overlook critical memory failure modes like forgetting.

Contribution

It proposes a new evaluation method that assesses memory evolution, generalization, and retention, providing deeper insights into LLM memory performance beyond final accuracy.

Findings

Higher accuracy does not always mean better memory quality.

Different memory methods show distinct trade-offs between adaptability and stability.

Many methods suffer from forgetting and negative transfer despite strong performance.

Abstract

Memory plays a central role in enabling large language models (LLMs) to operate over sequential tasks by accumulating and reusing experience over time. However, existing evaluations of LLM memory mostly rely on aggregate metrics such as final hold-out accuracy or cumulative online performance, which can obscure critical failure modes such as forgetting and negative transfer. In this paper, we introduce SeqMem-Eval, a diagnostic evaluation framework for sequentially evolving LLM memory. Drawing inspiration from continual learning, it targets a test-time setting in which memory is external, prompt-mediated, and updated without modifying model parameters. Rather than focusing only on final performance, SeqMem-Eval evaluates how memory states evolve, generalize, consolidate experience, and retain useful information during sequential inference. Specifically, it measures online utility, hold-out generalization, backward transfer, and forgetting, providing a finer-grained view of memory quality. Through extensive experiments across diverse tasks and memory methods, we show that higher final or cumulative accuracy does not necessarily imply better memory quality: many methods exhibit strong performance gains while suffering from substantial forgetting or negative transfer. Moreover, different memory designs exhibit distinct trade-offs between adaptability and stability that remain invisible under standard evaluation metrics.