MemGym: a Long-Horizon Memory Environment for LLM Agents

TL;DR

MemGym introduces a comprehensive benchmark for evaluating long-horizon memory in LLM agents across various agentic tasks, emphasizing memory formation and retention in realistic scenarios.

Contribution

The paper presents MemGym, a unified benchmark with synthetic pipelines and a reward model for evaluating memory in long-horizon LLM tasks, addressing limitations of existing benchmarks.

Findings

MemGym provides memory-isolated scores for better evaluation.

Synthetic pipelines are length-controllable and ablation-verified.

MemRM effectively scores coding environment memory quality.

Abstract

Memory is a central capability for LLM agents operating across long-horizon tasks. Existing memory benchmarks predominantly evaluate retention of personalized information in multi-turn chat scenarios, overlooking the dynamic memory formation that occurs during extended agent execution. Consequently, the memory systems they produce transfer poorly to realistic agentic environments, such as coding and web navigation. We present MemGym, a benchmark for agentic memory that unifies existing agent gyms and in-house memory-grounded pipelines behind one memory-reasoning interface. MemGym spans five evaluation tracks grouped into four agentic regimes: tool-use dialogue (tau2-bench), multi-turn deep-research search (MEMGYM-DR), coding (SWE-Gym and MEMGYM-CODEQA), and computer use (WebArena-Infinity). MemGym reports memory-isolated scores that decouple memory performance from reasoning, retrieval, and tool-use ability, so memory strategies can be ranked without those confounders. Our synthetic pipelines for MEMGYM-CODEQA and MEMGYM-DR are length-controllable, ablation-verified at every stage, and tightly aligned with downstream scenarios. To make evaluation on coding environments academically tractable, we train MemRM, a lightweight reward model (Qwen3-1.7B fine-tuned with QLoRA) that scores compression quality as a fast scalar read in place of full Docker rollouts.