MADial-Bench: Towards Real-world Evaluation of Memory-Augmented Dialogue Generation

TL;DR

This paper introduces MADial-Bench, a comprehensive benchmark for evaluating memory-augmented dialogue systems, emphasizing diverse memory recall and human-like response qualities beyond traditional metrics.

Contribution

It creates a novel benchmark based on cognitive science, incorporating new evaluation criteria for memory recall, emotion support, and intimacy in dialogue systems.

Findings

Embedding models show potential for improvement.

Memory injection correlates with emotion support.

Large language models perform well on the benchmark.

Abstract

Long-term memory is important for chatbots and dialogue systems (DS) to create consistent and human-like conversations, evidenced by numerous developed memory-augmented DS (MADS). To evaluate the effectiveness of such MADS, existing commonly used evaluation metrics, like retrieval accuracy and perplexity (PPL), mainly focus on query-oriented factualness and language quality assessment. However, these metrics often lack practical value. Moreover, the evaluation dimensions are insufficient for human-like assessment in DS. Regarding memory-recalling paradigms, current evaluation schemes only consider passive memory retrieval while ignoring diverse memory recall with rich triggering factors, e.g., emotions and surroundings, which can be essential in emotional support scenarios. To bridge the gap, we construct a novel Memory-Augmented Dialogue Benchmark (MADail-Bench) covering various memory-recalling paradigms based on cognitive science and psychology theories. The benchmark assesses two tasks separately: memory retrieval and memory recognition with the incorporation of both passive and proactive memory recall data. We introduce new scoring criteria to the evaluation, including memory injection, emotion support (ES) proficiency, and intimacy, to comprehensively assess generated responses. Results from cutting-edge embedding models and large language models on this benchmark indicate the potential for further advancement. Extensive testing further reveals correlations between memory injection, ES proficiency, and intimacy.