Human-inspired Episodic Memory for Infinite Context LLMs

TL;DR

This paper introduces EM-LLM, a human-inspired episodic memory system for large language models that enables handling of infinite contexts efficiently, improving coherence, accuracy, and retrieval over long sequences without fine-tuning.

Contribution

EM-LLM integrates human-like episodic memory mechanisms into LLMs, allowing for infinite context processing and efficient retrieval without additional training.

Findings

Outperforms state-of-the-art retrieval models like InfLLM and RAG across benchmarks.

Enables retrieval over 10 million tokens, surpassing full-context models in many tasks.

Shows strong correlation between event segmentation and human perception, indicating biological plausibility.

Abstract

Large language models (LLMs) have shown remarkable capabilities, but still struggle with processing extensive contexts, limiting their ability to maintain coherence and accuracy over long sequences. In contrast, the human brain excels at organising and retrieving episodic experiences across vast temporal scales, spanning a lifetime. In this work, we introduce EM-LLM, a novel approach that integrates key aspects of human episodic memory and event cognition into LLMs with no fine-tuning, enabling them to handle practically infinite context lengths while maintaining computational efficiency. EM-LLM organises sequences of tokens into coherent episodic events using a combination of Bayesian surprise and graph-theoretic boundary refinement in an online fashion. When needed, these events are retrieved through a two-stage memory process, combining similarity-based and temporally contiguous retrieval for efficient, human-inspired access to relevant information. Experiments on the LongBench and $\infty$-Bench benchmarks demonstrate EM-LLM's superior performance, consistently outperforming the state-of-the-art retrieval model InfLLM across various baseline LLMs. In addition, EM-LLM outperforms its popular counterpart, RAG, in a wide range of tasks, while requiring similar resources. Notably, EM-LLM's performance even surpasses full-context models in most tasks, while successfully performing retrieval across 10 million tokens -- a scale computationally infeasible for such models. Finally, our analysis reveals strong correlations between EM-LLM's event segmentation and human-perceived events, suggesting parallels between this artificial system and its biological counterpart, thereby offering a novel computational framework for exploring human memory mechanisms.