HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents

TL;DR

HiGMem introduces a hierarchical, LLM-guided memory system that improves long-term conversational agents by efficiently retrieving relevant memories using event summaries, reducing overhead and enhancing accuracy.

Contribution

The paper presents HiGMem, a novel two-level memory system that uses event summaries as semantic anchors for more precise and efficient memory retrieval in conversational agents.

Findings

Achieves the best F1 on four of five question categories in LoCoMo10.

Improves adversarial F1 from 0.54 to 0.78 over A-Mem.

Retrieves an order of magnitude fewer turns while maintaining performance.

Abstract

Long-term conversational large language model (LLM) agents require memory systems that can recover relevant evidence from historical interactions without overwhelming the answer stage with irrelevant context. However, existing memory systems, including hierarchical ones, still often rely solely on vector similarity for retrieval. It tends to produce bloated evidence sets: adding many superficially similar dialogue turns yields little additional recall, but lowers retrieval precision, increases answer-stage context cost, and makes retrieved memories harder to inspect and manage. To address this, we propose HiGMem (Hierarchical and LLM-Guided Memory System), a two-level event-turn memory system that allows LLMs to use event summaries as semantic anchors to predict which related turns are worth reading. This allows the model to inspect high-level event summaries first and then focus on a smaller set of potentially useful turns, providing a concise and reliable evidence set through reasoning, while avoiding the retrieval overhead that would be excessively high compared to vector retrieval. On the LoCoMo10 benchmark, HiGMem achieves the best F1 on four of five question categories and improves adversarial F1 from 0.54 to 0.78 over A-Mem, while retrieving an order of magnitude fewer turns. Code is publicly available at https://github.com/ZeroLoss-Lab/HiGMem.