StratMem-Bench: Evaluating Strategic Memory Use in Virtual Character Conversation Beyond Factual Recall

TL;DR

This paper introduces StratMem-Bench, a benchmark for evaluating how virtual characters strategically utilize memory in dialogues, highlighting current models' strengths and weaknesses in memory deployment beyond factual recall.

Contribution

The paper presents a new benchmark and evaluation framework for assessing strategic memory use in virtual character conversations, addressing a gap in existing memory utilization benchmarks.

Findings

Models excel at distinguishing required from irrelevant memories.

Models struggle with integrating supportive memories into dialogue decisions.

The benchmark reveals limitations in current large language models' strategic memory deployment.

Abstract

Achieving realistic human-like conversation for virtual characters requires not only a simple memorization and recall of past events, but also the strategic utilization of memory to meet factual needs and social engagement. Current memory utilization relevant (e.g., memory-augmented generation, long-term dialogue, and etc.) benchmarks overlook this nuance, treating memory primarily as a static repository of facts rather than a dynamic resource to be strategically deployed in dialogues. To address this gap, we design StratMem-Bench, a new benchmark to evaluate strategic memory use in character-centric dialogues. This dataset comprises 657 instances where virtual characters must navigate heterogeneous memory pools containing required, supportive, and irrelevant memories. We also propose a framework with different evaluation metrics including Strict Memory Compliance, Memory Integration Quality, Proactive Enrichment Score and Conditional Irrelevance Rate, to evaluate strategic memory use capabilities of virtual characters. Experiments on StratMem-Bench which leverage the state-of-the-art large language models as virtual characters show that all models perform well at distinguishing between required and irrelevant memories, but struggle once supportive memories are introduced into the decision process.