MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks

TL;DR

MemoryArena provides a comprehensive benchmark for evaluating how agents utilize long-term memory across interdependent, multi-session tasks, revealing gaps in current memory evaluation methods.

Contribution

It introduces MemoryArena, a unified benchmark for assessing agent memory in realistic multi-session, interdependent tasks, bridging a gap in existing evaluation frameworks.

Findings

Agents perform poorly on MemoryArena compared to existing benchmarks.

Memory utilization is crucial for solving interdependent, multi-session tasks.

Current benchmarks do not adequately evaluate memory in realistic agent settings.

Abstract

Existing evaluations of agents with memory typically assess memorization and action in isolation. One class of benchmarks evaluates memorization by testing recall of past conversations or text but fails to capture how memory is used to guide future decisions. Another class focuses on agents acting in single-session tasks without the need for long-term memory. However, in realistic settings, memorization and action are tightly coupled: agents acquire memory while interacting with the environment, and subsequently rely on that memory to solve future tasks. To capture this setting, we introduce MemoryArena, a unified evaluation gym for benchmarking agent memory in multi-session Memory-Agent-Environment loops. The benchmark consists of human-crafted agentic tasks with explicitly interdependent subtasks, where agents must learn from earlier actions and feedback by distilling experiences into memory, and subsequently use that memory to guide later actions to solve the overall task. MemoryArena supports evaluation across web navigation, preference-constrained planning, progressive information search, and sequential formal reasoning, and reveals that agents with near-saturated performance on existing long-context memory benchmarks like LoCoMo perform poorly in our agentic setting, exposing a gap in current evaluations for agents with memory.