EMP-EVAL: A Framework for Measuring Empathy in Open Domain Dialogues

TL;DR

This paper introduces EMP-EVAL, an automatic framework for measuring empathy in open-domain dialogues, aiming to reduce reliance on subjective human evaluations by capturing emotional and cognitive empathy components.

Contribution

It presents the first systematic method for automatic empathy measurement in conversations, incorporating emotional and cognitive aspects, with validation against human judgments.

Findings

Metrics correlate well with human preferences

Achieves comparable results to human judgments

First systematic approach for automatic empathy evaluation

Abstract

Measuring empathy in conversation can be challenging, as empathy is a complex and multifaceted psychological construct that involves both cognitive and emotional components. Human evaluations can be subjective, leading to inconsistent results. Therefore, there is a need for an automatic method for measuring empathy that reduces the need for human evaluations. In this paper, we proposed a novel approach EMP-EVAL, a simple yet effective automatic empathy evaluation method. The proposed technique takes the influence of Emotion, Cognitive and Emotional empathy. To the best knowledge, our work is the first to systematically measure empathy without the human-annotated provided scores. Experimental results demonstrate that our metrics can correlate with human preference, achieving comparable results with human judgments.