Temporal Modeling Matters: A Novel Temporal Emotional Modeling Approach for Speech Emotion Recognition

TL;DR

This paper introduces TIM-Net, a novel multi-scale temporal modeling approach for speech emotion recognition that effectively captures emotional dynamics across various time scales, leading to improved recognition accuracy.

Contribution

The paper proposes TIM-Net, a new temporal-aware multi-scale network that models speech emotions across different time scales, enhancing the ability to recognize dynamic emotional states.

Findings

TIM-Net outperforms existing methods on six benchmark datasets.

Achieves 2.34% and 2.61% improvements in UAR and WAR respectively.

Demonstrates the effectiveness of multi-scale temporal modeling in SER.

Abstract

Speech emotion recognition (SER) plays a vital role in improving the interactions between humans and machines by inferring human emotion and affective states from speech signals. Whereas recent works primarily focus on mining spatiotemporal information from hand-crafted features, we explore how to model the temporal patterns of speech emotions from dynamic temporal scales. Towards that goal, we introduce a novel temporal emotional modeling approach for SER, termed Temporal-aware bI-direction Multi-scale Network (TIM-Net), which learns multi-scale contextual affective representations from various time scales. Specifically, TIM-Net first employs temporal-aware blocks to learn temporal affective representation, then integrates complementary information from the past and the future to enrich contextual representations, and finally, fuses multiple time scale features for better adaptation to the emotional variation. Extensive experimental results on six benchmark SER datasets demonstrate the superior performance of TIM-Net, gaining 2.34% and 2.61% improvements of the average UAR and WAR over the second-best on each corpus. The source code is available at https://github.com/Jiaxin-Ye/TIM-Net_SER.