Amory: Building Coherent Narrative-Driven Agent Memory through Agentic Reasoning

TL;DR

Amory introduces a memory framework for conversational agents that constructs structured, narrative-based memories through agentic reasoning, improving long-term coherence and efficiency over existing methods.

Contribution

It presents a novel memory system that actively organizes conversations into narratives and consolidates memories, enhancing coherence and reducing computational costs.

Findings

Achieves comparable performance to full context reasoning with 50% faster response times.

Momentum-aware consolidation improves response quality.

Coherence-driven retrieval outperforms embedding-based approaches in memory coverage.

Abstract

Long-term conversational agents face a fundamental scalability challenge as interactions extend over time: repeatedly processing entire conversation histories becomes computationally prohibitive. Current approaches attempt to solve this through memory frameworks that predominantly fragment conversations into isolated embeddings or graph representations and retrieve relevant ones in a RAG style. While computationally efficient, these methods often treat memory formation minimally and fail to capture the subtlety and coherence of human memory. We introduce Amory, a working memory framework that actively constructs structured memory representations through enhancing agentic reasoning during offline time. Amory organizes conversational fragments into episodic narratives, consolidates memories with momentum, and semanticizes peripheral facts into semantic memory. At retrieval time, the system employs coherence-driven reasoning over narrative structures. Evaluated on the LOCOMO benchmark for long-term reasoning, Amory achieves considerable improvements over previous state-of-the-art, with performance comparable to full context reasoning while reducing response time by 50%. Analysis shows that momentum-aware consolidation significantly enhances response quality, while coherence-driven retrieval provides superior memory coverage compared to embedding-based approaches.