EngramaBench: Evaluating Long-Term Conversational Memory with Structured Graph Retrieval

TL;DR

EngramaBench is a new benchmark designed to evaluate long-term conversational memory in AI, comparing different memory architectures like graph-structured memory and vector retrieval using GPT-4o.

Contribution

The paper introduces EngramaBench, a comprehensive benchmark for long-term memory in conversational AI, and evaluates a novel graph-structured memory system against existing methods.

Findings

GPT-4o full-context performs best overall.

Engrama outperforms in cross-space reasoning.

Mem0 is the most cost-effective but less accurate.

Abstract

Large language model assistants are increasingly expected to retain and reason over information accumulated across many sessions. We introduce EngramaBench, a benchmark for long-term conversational memory built around five personas, one hundred multi-session conversations, and one hundred fifty queries spanning factual recall, cross-space integration, temporal reasoning, adversarial abstention, and emergent synthesis. We evaluate Engrama, a graph-structured memory system, against GPT-4o full-context prompting and Mem0, an open-source vector-retrieval memory system. All three use the same answering model (GPT-4o), isolating the effect of memory architecture. GPT-4o full-context achieves the highest composite score (0.6186), while Engrama scores 0.5367 globally but is the only system to score higher than full-context prompting on cross-space reasoning (0.6532 vs. 0.6291, n=30). Mem0 is cheapest but substantially weaker (0.4809). Ablations reveal that the components driving Engrama's cross-space advantage trade off against global composite score, exposing a systems-level tension between structured memory specialization and aggregate optimization.