Reflecting Twice before Speaking with Empathy: Self-Reflective Alternating Inference for Empathy-Aware End-to-End Spoken Dialogue

TL;DR

This paper introduces ReEmpathy, an end-to-end spoken language model that improves empathetic dialogue by incorporating a self-reflective inference mechanism, addressing limitations of traditional supervised training methods.

Contribution

It proposes a novel Empathetic Self-Reflective Alternating Inference mechanism and the EmpathyEval evaluation model, advancing empathy modeling in spoken dialogue systems.

Findings

ReEmpathy outperforms baseline models in empathetic response quality.

Reflective reasoning enhances emotional intelligence in dialogue systems.

Empathy-aware dialogue systems show significant improvement in experiments.

Abstract

End-to-end Spoken Language Models (SLMs) hold great potential for paralinguistic perception, and numerous studies have aimed to enhance their capabilities, particularly for empathetic dialogue. However, current approaches largely depend on rigid supervised signals, such as ground-truth response in supervised fine-tuning or preference scores in reinforcement learning. Such reliance is fundamentally limited for modeling complex empathy, as there is no single "correct" response and a simple numerical score cannot fully capture the nuances of emotional expression or the appropriateness of empathetic behavior. To address these limitations, we sequentially introduce EmpathyEval, a descriptive natural-language-based evaluation model for assessing empathetic quality in spoken dialogues. Building upon EmpathyEval, we propose ReEmpathy, an end-to-end SLM that enhances empathetic dialogue through a novel Empathetic Self-Reflective Alternating Inference mechanism, which interleaves spoken response generation with free-form, empathy-related reflective reasoning. Extensive experiments demonstrate that ReEmpathy substantially improves empathy-sensitive spoken dialogue by enabling reflective reasoning, offering a promising approach toward more emotionally intelligent and empathy-aware human-computer interactions.