What Deserves Memory: Adaptive Memory Distillation for LLM Agents

TL;DR

NEMORI introduces an adaptive, data-driven memory distillation framework for LLM agents that predicts future utility based on sequence predictability, improving efficiency and storage.

Contribution

The paper presents NEMORI, a novel memory system that uses predictability for experience retention, moving beyond heuristic-based approaches.

Findings

NEMORI achieves strong performance in memory retention tasks.

The framework reduces storage requirements significantly.

Experiments demonstrate improved efficiency over existing methods.

Abstract

Memory systems for LLM agents struggle to determine what information deserves retention. Existing approaches rely on predefined heuristics such as importance scores, emotional tags, or factual templates, encoding designer intuition rather than learning from the data itself. Inspired by cognitive ideas, we propose NEMORI, an adaptive memory distillation framework that casts the assessment of experience's future utility as a matter of predictability. Specifically, NEMORI comprises two cascading modules: Episodic Memory Integration transforms raw interactions into coherent narratives, and Semantic Knowledge Distillation extracts insights via prediction error. Centering on distillation, the framework remains agnostic to downstream management. Extensive experiments confirm that NEMORI achieves strong performance, efficiency, and storage reduction. Our work suggests that observing the intrinsic properties of interaction sequences offers a viable, data-driven alternative to heuristic-based memory design. Code: https://github.com/nemori-ai/nemori.