Empathy Omni: Enabling Empathetic Speech Response Generation through Large Language Models

TL;DR

Emotion Omni is a novel speech LLM that understands emotional cues in user speech and generates empathetic responses, achieving high speech quality and emotional expressiveness with limited data and without large-scale pretraining.

Contribution

The paper introduces Emotion Omni, a model capable of empathetic speech response generation using a new emotional dialogue dataset and a data pipeline, reducing reliance on massive datasets.

Findings

Emotion Omni achieves high speech quality (UTMOS:4.41).

It surpasses existing models in empathy (Emotion GPT Score: 3.97).

It maintains instruction-following ability without large-scale pretraining.

Abstract

With the development of speech large language models (speech LLMs), users can now interact directly with assistants via speech. However, most existing models only convert response content into speech without fully capturing the rich emotional cues in user queries, where the same sentence may convey different meanings depending on the expression. Emotional understanding is thus essential for improving human-machine interaction. Most empathetic speech LLMs rely on massive datasets, demanding high computational cost. A key challenge is to build models that generate empathetic responses with limited data and without large-scale training. To this end, we propose Emotion Omni, a model that understands emotional content in user speech and generates empathetic responses. We further developed a data pipeline to construct a 200k emotional dialogue dataset supporting empathetic speech assistants. Experiments show that Emotion Omni achieves comparable instruction-following ability without large-scale pretraining, while surpassing existing models in speech quality (UTMOS:4.41) and empathy (Emotion GPT Score: 3.97). These results confirm its improvements in both speech fidelity and emotional expressiveness. Demos are available at https://w311411.github.io/omni_demo/.