MemConflict: Evaluating Long-Term Memory Systems Under Memory Conflicts

TL;DR

MemConflict introduces a diagnostic framework to evaluate long-term memory systems in language models, focusing on retrieval accuracy and conflict resolution across multi-session interactions.

Contribution

It formalizes conflict types in memory retrieval, creates a benchmark with controlled conflicts, and provides insights into system strengths and weaknesses.

Findings

Memory correctness often diverges from retrieval and ranking.

Longer histories and distractors degrade performance.

Failures include missing supporting memories and ineffective retrieval.

Abstract

Long-term memory systems enable conversational agents based on large language models (LLMs) to retain, retrieve, and apply user-specific information across multi-session interactions. However, existing evaluations mainly assess outcome-level performance or temporal updating, providing limited insight into how systems retrieve and rank temporally valid, factually correct, and contextually applicable memory evidence under conflicting alternatives. To address this gap, we propose MemConflict, a diagnostic framework that treats memory validity as a query-conditioned fitness-for-use problem. MemConflict formalizes dynamic, static, and conditional conflicts over temporal validity, factual correctness, and contextual applicability. It simulates controlled long-horizon histories from structured user profiles, introduces cross-session conflicts, and injects semantically similar distractors to create competition among memory candidates. The resulting multi-session dialogue benchmark supports black-box evaluation of final answers and white-box analysis of supporting-memory retrieval and ranking. Experiments on six representative long-term memory systems show uneven strengths across conflict types, with answer correctness often diverging from memory retrieval and ranking. Sensitivity analyses reveal that longer histories, distractors, implicit queries, and larger conflict distances degrade performance. Diagnostics show failures from missing supporting memories and ineffective use of retrieved memories. Collectively, MemConflict advances principled long-term memory governance through retrieval-aware, conflict-aware reliability assessment.