REMem: Reasoning with Episodic Memory in Language Agent

TL;DR

REMem introduces a novel episodic memory framework for language agents, enabling effective recollection and reasoning over interaction histories, significantly outperforming existing memory systems on multiple benchmarks.

Contribution

It formalizes the core challenges of episodic memory in language agents and proposes a two-phase framework with hybrid memory graphs and iterative retrieval for improved reasoning.

Findings

Outperforms state-of-the-art memory systems by 3.4% and 13.4% on key tasks.

Demonstrates robust refusal behavior for unanswerable questions.

Achieves significant improvements across four episodic memory benchmarks.

Abstract

Humans excel at remembering concrete experiences along spatiotemporal contexts and performing reasoning across those events, i.e., the capacity for episodic memory. In contrast, memory in language agents remains mainly semantic, and current agents are not yet capable of effectively recollecting and reasoning over interaction histories. We identify and formalize the core challenges of episodic recollection and reasoning from this gap, and observe that existing work often overlooks episodicity, lacks explicit event modeling, or overemphasizes simple retrieval rather than complex reasoning. We present REMem, a two-phase framework for constructing and reasoning with episodic memory: 1) Offline indexing, where REMem converts experiences into a hybrid memory graph that flexibly links time-aware gists and facts. 2) Online inference, where REMem employs an agentic retriever with carefully curated tools for iterative retrieval over the memory graph. Comprehensive evaluation across four episodic memory benchmarks shows that REMem substantially outperforms state-of-the-art memory systems such as Mem0 and HippoRAG 2, showing 3.4% and 13.4% absolute improvements on episodic recollection and reasoning tasks, respectively. Moreover, REMem also demonstrates more robust refusal behavior for unanswerable questions.