A Unified Spoken Language Model with Injected Emotional-Attribution Thinking for Human-like Interaction

TL;DR

This paper introduces a unified spoken language model with emotional reasoning capabilities, utilizing a novel IEAT strategy to incorporate emotional states and causes, resulting in superior empathetic dialogue performance.

Contribution

The paper proposes Injected Emotional-Attribution Thinking (IEAT), a new data construction method that enhances emotional reasoning in spoken language models through a two-stage training process.

Findings

Achieves top performance on HumDial Emotional Intelligence benchmark.

Demonstrates effective emotion-aware reasoning and empathetic response generation.

Outperforms existing models in emotional trajectory modeling.

Abstract

This paper presents a unified spoken language model for emotional intelligence, enhanced by a novel data construction strategy termed Injected Emotional-Attribution Thinking (IEAT). IEAT incorporates user emotional states and their underlying causes into the model's internal reasoning process, enabling emotion-aware reasoning to be internalized rather than treated as explicit supervision. The model is trained with a two-stage progressive strategy. The first stage performs speech-text alignment and emotional attribute modeling via self-distillation, while the second stage conducts end-to-end cross-modal joint optimization to ensure consistency between textual and spoken emotional expressions. Experiments on the Human-like Spoken Dialogue Systems Challenge (HumDial) Emotional Intelligence benchmark demonstrate that the proposed approach achieves top-ranked performance across emotional trajectory modeling, emotional reasoning, and empathetic response generation under both LLM-based and human evaluations.