Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations

TL;DR

This paper provides a structured taxonomy and empirical analysis of agentic memory systems in large language models, highlighting current limitations and proposing directions for improved evaluation and scalability.

Contribution

It introduces a taxonomy of MAG systems, analyzes key empirical limitations, and connects memory structures to performance issues, guiding future research.

Findings

Benchmark saturation affects evaluation reliability.

Performance varies significantly across backbone models.

Memory maintenance incurs latency and throughput costs.

Abstract

Agentic memory systems enable large language model (LLM) agents to maintain state across long interactions, supporting long-horizon reasoning and personalization beyond fixed context windows. Despite rapid architectural development, the empirical foundations of these systems remain fragile: existing benchmarks are often underscaled, evaluation metrics are misaligned with semantic utility, performance varies significantly across backbone models, and system-level costs are frequently overlooked. This survey presents a structured analysis of agentic memory from both architectural and system perspectives. We first introduce a concise taxonomy of MAG systems based on four memory structures. Then, we analyze key pain points limiting current systems, including benchmark saturation effects, metric validity and judge sensitivity, backbone-dependent accuracy, and the latency and throughput overhead introduced by memory maintenance. By connecting the memory structure to empirical limitations, this survey clarifies why current agentic memory systems often underperform their theoretical promise and outlines directions for more reliable evaluation and scalable system design.