Back to Basics: Let Conversational Agents Remember with Just Retrieval and Generation

TL;DR

This paper introduces extbackslash method, a minimalist conversational memory framework using retrieval and generation, addressing issues of context dilution by focusing on turn-level signals and redundancy removal.

Contribution

It proposes a simple yet effective approach that replaces complex memory architectures with retrieval and pruning techniques, improving robustness and efficiency.

Findings

extbackslash method outperforms strong baselines on multiple benchmarks.

It maintains high efficiency in tokens and latency.

It establishes a new minimalist baseline for conversational memory.

Abstract

Existing conversational memory systems rely on complex hierarchical summarization or reinforcement learning to manage long-term dialogue history, yet remain vulnerable to context dilution as conversations grow. In this work, we offer a different perspective: the primary bottleneck may lie not in memory architecture, but in the \textit{Signal Sparsity Effect} within the latent knowledge manifold. Through controlled experiments, we identify two key phenomena: \textit{Decisive Evidence Sparsity}, where relevant signals become increasingly isolated with longer sessions, leading to sharp degradation in aggregation-based methods; and \textit{Dual-Level Redundancy}, where both inter-session interference and intra-session conversational filler introduce large amounts of non-informative content, hindering effective generation. Motivated by these insights, we propose \method, a minimalist framework that brings conversational memory back to basics, relying solely on retrieval and generation via Turn Isolation Retrieval (TIR) and Query-Driven Pruning (QDP). TIR replaces global aggregation with a max-activation strategy to capture turn-level signals, while QDP removes redundant sessions and conversational filler to construct a compact, high-density evidence set. Extensive experiments on multiple benchmarks demonstrate that \method achieves robust performance across diverse settings, consistently outperforming strong baselines while maintaining high efficiency in tokens and latency, establishing a new minimalist baseline for conversational memory.