MedMemoryBench: Benchmarking Agent Memory in Personalized Healthcare

TL;DR

MedMemoryBench introduces a realistic benchmarking framework for medical agent memory, emphasizing long-term clinical tracking, dynamic evaluation, and exposing bottlenecks in current architectures for healthcare applications.

Contribution

It presents a novel benchmark with a realistic, long-horizon medical dataset and streaming evaluation protocol, highlighting critical memory saturation issues in existing models.

Findings

Severe bottlenecks identified in mainstream architectures for medical reasoning.

Memory saturation degrades retrieval and reasoning robustness.

Benchmark facilitates development of more robust medical agents.

Abstract

The large-scale deployment of personalized healthcare agents demands memory mechanisms that are exceptionally precise, safe, and capable of long-term clinical tracking. However, existing benchmarks primarily focus on daily open-domain conversations, failing to capture the high-stakes complexity of real-world medical applications. Motivated by the stringent production requirements of an industry-leading health management agent serving tens of millions of active users, we introduce MedMemoryBench. We develop a human-agent collaborative pipeline to synthesize highly realistic, long-horizon medical trajectories based on clinically grounded, synthetic patient archetypes. This process yields a massive, expertly validated dataset comprising approximately 2,000 sessions and 16,000 interaction turns. Crucially, MedMemoryBench departs from traditional static evaluations by pioneering an "evaluate-while-constructing" streaming assessment protocol, which precisely mirrors dynamic memory accumulation in production environments. Furthermore, we formalize and systematically investigate the critical phenomenon of memory saturation, where sustained information influx actively degrades retrieval and reasoning robustness. Comprehensive benchmarking reveals severe bottlenecks in mainstream architectures, particularly concerning complex medical reasoning and noise resilience. By exposing these fundamental flaws, MedMemoryBench establishes a vital foundation for developing robust, production-ready medical agents.