Incorporating Contextual Paralinguistic Understanding in Large Speech-Language Models

TL;DR

This paper introduces methods to enhance large speech-language models with contextual paralinguistic cues, significantly improving their empathetic reasoning and understanding of emotional content in speech.

Contribution

It proposes explicit and implicit approaches to incorporate paralinguistic information into Speech-LLMs, boosting performance on emotion-related tasks.

Findings

Implicit method improves QA benchmark performance by 38.41%.

Combined approaches reach 46.02% performance.

LLM judge correlates well with classification metrics.

Abstract

Current large speech language models (Speech-LLMs) often exhibit limitations in empathetic reasoning, primarily due to the absence of training datasets that integrate both contextual content and paralinguistic cues. In this work, we propose two approaches to incorporate contextual paralinguistic information into model training: (1) an explicit method that provides paralinguistic metadata (e.g., emotion annotations) directly to the LLM, and (2) an implicit method that automatically generates novel training question-answer (QA) pairs using both categorical and dimensional emotion annotations alongside speech transcriptions. Our implicit method boosts performance (LLM-judged) by 38.41% on a human-annotated QA benchmark, reaching 46.02% when combined with the explicit approach, showing effectiveness in contextual paralinguistic understanding. We also validate the LLM judge by demonstrating its correlation with classification metrics, providing support for its reliability.