Evaluating Emotion Recognition in Spoken Language Models on Emotionally Incongruent Speech

TL;DR

This paper evaluates how well spoken language models recognize emotions in speech when the semantic content and speech expressiveness are incongruent, revealing a reliance on textual semantics over acoustic cues.

Contribution

It introduces an evaluation of SLMs on emotionally incongruent speech and releases a new dataset and code for further research.

Findings

SLMs rely mainly on textual semantics for emotion recognition.

Speech expressiveness has limited influence on model predictions.

The study highlights the need for better integration of audio and text modalities.

Abstract

Advancements in spoken language processing have driven the development of spoken language models (SLMs), designed to achieve universal audio understanding by jointly learning text and audio representations for a wide range of tasks. Although promising results have been achieved, there is growing discussion regarding these models' generalization capabilities and the extent to which they truly integrate audio and text modalities in their internal representations. In this work, we evaluate four SLMs on the task of speech emotion recognition using a dataset of emotionally incongruent speech samples, a condition under which the semantic content of the spoken utterance conveys one emotion while speech expressiveness conveys another. Our results indicate that SLMs rely predominantly on textual semantics rather than speech emotion to perform the task, indicating that text-related representations largely dominate over acoustic representations. We release both the code and the Emotionally Incongruent Synthetic Speech dataset (EMIS) to the community.