MoodBench 1.0: An Evaluation Benchmark for Emotional Companionship Dialogue Systems

TL;DR

MoodBench 1.0 is a comprehensive evaluation benchmark designed to systematically assess the emotional companionship capabilities of dialogue systems, addressing a key gap in the development of emotionally supportive AI.

Contribution

This paper introduces the first formal definition of Emotional Companionship Dialogue Systems and presents MoodBench 1.0, a benchmark for evaluating their emotional support abilities.

Findings

MoodBench 1.0 effectively differentiates models' emotional capabilities.

Current models show significant room for improvement in deep emotional support.

The benchmark guides future development of more emotionally intelligent dialogue systems.

Abstract

With the rapid development of Large Language Models, dialogue systems are shifting from information tools to emotional companions, heralding the era of Emotional Companionship Dialogue Systems (ECDs) that provide personalized emotional support for users. However, the field lacks clear definitions and systematic evaluation standards for ECDs. To address this, we first propose a definition of ECDs with formal descriptions. Then, based on this theory and the design principle of "Ability Layer-Task Layer (three level)-Data Layer-Method Layer", we design and implement the first ECD evaluation benchmark - MoodBench 1.0. Through extensive evaluations of 30 mainstream models, we demonstrate that MoodBench 1.0 has excellent discriminant validity and can effectively quantify the differences in emotional companionship abilities among models. Furthermore, the results reveal current models' shortcomings in deep emotional companionship, guiding future technological optimization and significantly aiding developers in enhancing ECDs' user experience.