Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation

TL;DR

This paper introduces xMemory, a hierarchical memory structure for agent systems that improves retrieval efficiency and answer quality by decoupling and aggregating relevant information before retrieval.

Contribution

It proposes a novel decoupling and aggregation principle for agent memory, leading to the development of xMemory, which enhances retrieval and reasoning in LLM-based agents.

Findings

xMemory improves answer quality across multiple datasets.

It reduces redundancy and increases evidence density in retrieved information.

xMemory achieves higher inference token efficiency.

Abstract

Standard Retrieval Augmented Generation (RAG) is poorly matched to agent memory. Unlike large heterogeneous corpora, agent memory forms a bounded and coherent interaction stream in which many spans are highly correlated or near duplicates. As a result, flat top-$k$ similarity retrieval often returns redundant context, while summary-centric hierarchies can blur the subtle details that distinguish one candidate from another. We argue that agent memory should follow the principle of decoupling before aggregation: the system should first isolate reusable facts, updates, and distinguishing details from similar histories, and only then organise them for efficient retrieval. Based on this principle, we propose xMemory, which constructs a revisable hierarchical memory structure from original messages to segments, memory components, and groups. xMemory segments interaction history into local events, decouples each segment into memory components, aggregates related components into high-level groups using a sparsity--semantic faithfulness objective, and maintains this structure incrementally as memory evolves. At inference time, xMemory retrieves top-down, first selecting a compact backbone of complementary groups and components, and then expanding to segments and raw messages only when additional evidence reduces the reader's uncertainty. Experiments on LoCoMo and PerLTQA across diverse open source and closed source LLMs show consistent gains in answer quality and inference token efficiency, supported by analyses of redundancy, evidence density, and coverage.