MemGround: Long-Term Memory Evaluation Kit for Large Language Models in Gamified Scenarios

TL;DR

MemGround introduces a comprehensive benchmark for evaluating long-term memory in large language models within gamified, interactive scenarios, addressing limitations of static evaluation methods.

Contribution

It proposes a hierarchical framework and multi-dimensional metrics for assessing dynamic memory capabilities in LLMs during complex interactions.

Findings

State-of-the-art LLMs struggle with sustained dynamic tracking.

Models have difficulty with temporal event association.

Complex reasoning from long-term evidence remains challenging.

Abstract

Current evaluations of long-term memory in LLMs are fundamentally static. By fixating on simple retrieval and short-context inference, they neglect the multifaceted nature of complex memory systems, such as dynamic state tracking and hierarchical reasoning in continuous interactions. To overcome these limitations, we propose MemGround, a rigorous long-term memory benchmark natively grounded in rich, gamified interactive scenarios. To systematically assess these capabilities, MemGround introduces a three-tier hierarchical framework that evaluates Surface State Memory, Temporal Associative Memory, and Reasoning-Based Memory through specialized interactive tasks. Furthermore, to comprehensively quantify both memory utilization and behavioral trajectories, we propose a multi-dimensional metric suite comprising Question-Answer Score (QA Overall), Memory Fragments Unlocked (MFU), Memory Fragments with Correct Order (MFCO), and Exploration Trajectory Diagrams (ETD). Extensive experiments reveal that state-of-the-art LLMs and memory agents still struggle with sustained dynamic tracking, temporal event association, and complex reasoning derived from long-term accumulated evidence in interactive environments.