Multi-dimensional Evaluation of Empathetic Dialog Responses

TL;DR

This paper introduces a multi-dimensional framework for evaluating conversational empathy, considering both expressed speaker intents and perceived listener empathy, and explores automated measurement methods with mixed results.

Contribution

It proposes a novel multi-dimensional empathy evaluation framework and compares different automated measurement techniques, highlighting the effectiveness of instruction-finetuned classifiers.

Findings

Perceived empathy correlates strongly with dialogue satisfaction.

Prompting LLMs like GPT-4 perform poorly in empathy measurement.

Instruction-finetuned classifiers outperform prior methods.

Abstract

Empathy is critical for effective and satisfactory conversational communication. Prior efforts to measure conversational empathy mostly focus on expressed communicative intents -- that is, the way empathy is expressed. Yet, these works ignore the fact that conversation is also a collaboration involving both speakers and listeners. In contrast, we propose a multi-dimensional empathy evaluation framework to measure both \emph{expressed intents from the speaker's perspective} and \emph{perceived empathy from the listener's perspective}. We apply our analytical framework to examine internal customer-service dialogues. We find the two dimensions (expressed intent types and perceived empathy) are inter-connected, while perceived empathy has high correlations with dialogue satisfaction levels. To reduce the annotation cost, we explore different options to automatically measure conversational empathy: prompting LLMs and training language model-based classifiers. Our experiments show that prompting methods with even popular models like GPT-4 and Flan family models perform relatively poorly on both public and our internal datasets. In contrast, instruction-finetuned classifiers based on Flan-T5 family models outperform prior works and competitive baselines. We conduct a detailed ablation study to give more insights into instruction finetuning method's strong performance.