Evaluating Memory Structure in LLM Agents

TL;DR

This paper introduces StructMemEval, a benchmark to evaluate how well LLM-based agents can organize and utilize complex long-term memory structures, revealing current limitations and guiding future improvements.

Contribution

It proposes a new benchmark for testing complex memory organization in LLM agents, addressing a gap in existing factual recall benchmarks.

Findings

Simple retrieval-augmented LLMs struggle with structured memory tasks.

Memory agents can solve tasks when prompted on memory organization.

Modern LLMs often fail to recognize memory structures without explicit prompts.

Abstract

Modern LLM-based agents and chat assistants rely on long-term memory frameworks to store reusable knowledge, recall user preferences, and augment reasoning. As researchers create more complex memory architectures, it becomes increasingly difficult to analyze their capabilities and guide future memory designs. Most long-term memory benchmarks focus on simple fact retention, multi-hop recall, and time-based changes. While undoubtedly important, these capabilities can often be achieved with simple retrieval-augmented LLMs and do not test complex memory hierarchies. To bridge this gap, we propose StructMemEval - a benchmark that tests the agent's ability to organize its long-term memory, not just factual recall. We gather a suite of tasks that humans solve by organizing their knowledge in a specific structure: transaction ledgers, to-do lists, trees and others. Our initial experiments show that simple retrieval-augmented LLMs struggle with these tasks, whereas memory agents can reliably solve them if prompted how to organize their memory. However, we also find that modern LLMs do not always recognize the memory structure when not prompted to do so. This highlights an important direction for future improvements in both LLM training and memory frameworks.