When Tone and Words Disagree: Towards Robust Speech Emotion Recognition under Acoustic-Semantic Conflict

TL;DR

This paper addresses the challenge of acoustic-semantic conflicts in speech emotion recognition by proposing a novel framework that disentangles acoustic and semantic information, and introduces a new dataset for evaluation.

Contribution

The paper introduces the Fusion Acoustic-Semantic (FAS) framework and the CASE dataset, improving SER robustness under conflicting acoustic and semantic cues.

Findings

FAS outperforms existing methods in various settings.

Conventional SER models fail under acoustic-semantic conflicts.

FAS achieves 59.38% accuracy on the CASE benchmark.

Abstract

Speech Emotion Recognition (SER) systems often assume congruence between vocal emotion and lexical semantics. However, in real-world interactions, acoustic-semantic conflict is common yet overlooked, where the emotion conveyed by tone contradicts the literal meaning of spoken words. We show that state-of-the-art SER models, including ASR-based, self-supervised learning (SSL) approaches and Audio Language Models (ALMs), suffer performance degradation under such conflicts due to semantic bias or entangled acoustic-semantic representations. To address this, we propose the Fusion Acoustic-Semantic (FAS) framework, which explicitly disentangles acoustic and semantic pathways and bridges them through a lightweight, query-based attention module. To enable systematic evaluation, we introduce the Conflict in Acoustic-Semantic Emotion (CASE), the first dataset dominated by clear and interpretable acoustic-semantic conflicts in varied scenarios. Extensive experiments demonstrate that FAS consistently outperforms existing methods in both in-domain and zero-shot settings. Notably, on the CASE benchmark, conventional SER models fail dramatically, while FAS sets a new SOTA with 59.38% accuracy. Our code and datasets is available at https://github.com/24DavidHuang/FAS.