Temporal-Frequency State Space Duality: An Efficient Paradigm for Speech Emotion Recognition

TL;DR

This paper introduces TF-Mamba, a multi-domain framework that effectively captures emotional cues in both temporal and frequency domains for speech emotion recognition, improving efficiency and robustness.

Contribution

The paper presents a novel temporal-frequency mamba block and a Complex Metric-Distance Triplet loss, enhancing emotion recognition by leveraging dual-domain features with improved efficiency.

Findings

Outperforms existing methods on IEMOCAP and MELD datasets.

Reduces model size and latency compared to prior approaches.

Achieves a better balance between computational efficiency and model expressiveness.

Abstract

Speech Emotion Recognition (SER) plays a critical role in enhancing user experience within human-computer interaction. However, existing methods are overwhelmed by temporal domain analysis, overlooking the valuable envelope structures of the frequency domain that are equally important for robust emotion recognition. To overcome this limitation, we propose TF-Mamba, a novel multi-domain framework that captures emotional expressions in both temporal and frequency dimensions.Concretely, we propose a temporal-frequency mamba block to extract temporal- and frequency-aware emotional features, achieving an optimal balance between computational efficiency and model expressiveness. Besides, we design a Complex Metric-Distance Triplet (CMDT) loss to enable the model to capture representative emotional clues for SER. Extensive experiments on the IEMOCAP and MELD datasets show that TF-Mamba surpasses existing methods in terms of model size and latency, providing a more practical solution for future SER applications.