From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents

TL;DR

This paper introduces Memora, a comprehensive benchmark for evaluating long-term memory in personalized agents over weeks to months, highlighting current shortcomings in memory management and updating.

Contribution

It presents Memora, a new benchmark with automated and human quality checks, and a novel Forgetting-Aware Memory Accuracy metric to better assess long-term memory performance.

Findings

LLMs and memory agents often reuse invalid memories.

Memory agents show only marginal improvements over baseline.

Agents struggle to reconcile evolving memories over time.

Abstract

Personalized agents that interact with users over long periods must maintain persistent memory across sessions and update it as circumstances change. However, existing benchmarks predominantly frame long-term memory evaluation as fact retrieval from past conversations, providing limited insight into agents' ability to consolidate memory over time or handle frequent knowledge updates. We introduce Memora, a long-term memory benchmark spanning weeks to months long user conversations. The benchmark evaluates three memory-grounded tasks: remembering, reasoning, and recommending. To ensure data quality, we employ automated memory-grounding checks and human evaluation. We further introduce Forgetting-Aware Memory Accuracy (FAMA), a metric that penalizes reliance on obsolete or invalidated memory when evaluating long-term memory. Evaluations of four LLMs and six memory agents reveal frequent reuse of invalid memories and failures to reconcile evolving memories. Memory agents offer marginal improvements, exposing shortcomings in long-term memory for personalized agents.