NextMem: Towards Latent Factual Memory for LLM-based Agents

TL;DR

NextMem introduces a latent factual memory framework for LLM-based agents, utilizing an autoregressive autoencoder with a two-stage training process and quantization to improve efficiency, accuracy, and robustness in memory tasks.

Contribution

The paper presents NextMem, a novel latent factual memory approach that overcomes limitations of existing methods through innovative training and compression techniques.

Findings

Outperforms existing methods in retrieval accuracy

Demonstrates robustness against memory degradation

Shows extensibility for larger-scale memory tasks

Abstract

Memory is critical for LLM-based agents to preserve past observations for future decision-making, where factual memory serves as its foundational part. However, existing approaches to constructing factual memory face several limitations. Textual methods impose heavy context and indexing burdens, while parametric methods suffer from catastrophic forgetting and high costs. To address these challenges, we introduce NextMem, a latent factual memory framework that utilizes an autoregressive autoencoder to efficiently construct latent memory while ensuring accurate reconstruction. For better optimization, we propose a two-stage training process, including autoregressive reconstruction alignment and progressive latent substitution. We also incorporate quantization to reduce storage overhead. Extensive experiments demonstrate that NextMem achieves superior performance, and excels in retrieval, robustness, and extensibility properties. We release our code and model checkpoints at https://github.com/nuster1128/NextMem.