Time-Frequency Transformer: A Novel Time Frequency Joint Learning Method for Speech Emotion Recognition

TL;DR

This paper introduces a novel Time-Frequency Transformer model that jointly learns time and frequency domain features for speech emotion recognition, capturing both local and global emotional patterns.

Contribution

It proposes a new joint learning framework using Transformer models to effectively model local and global emotional features in speech signals.

Findings

Outperforms state-of-the-art methods on IEMOCAP and CASIA datasets.

Effectively captures global emotion patterns in time-frequency domain.

Models local emotional correlations in time and frequency domains.

Abstract

In this paper, we propose a novel time-frequency joint learning method for speech emotion recognition, called Time-Frequency Transformer. Its advantage is that the Time-Frequency Transformer can excavate global emotion patterns in the time-frequency domain of speech signal while modeling the local emotional correlations in the time domain and frequency domain respectively. For the purpose, we first design a Time Transformer and Frequency Transformer to capture the local emotion patterns between frames and inside frequency bands respectively, so as to ensure the integrity of the emotion information modeling in both time and frequency domains. Then, a Time-Frequency Transformer is proposed to mine the time-frequency emotional correlations through the local time-domain and frequency-domain emotion features for learning more discriminative global speech emotion representation. The whole process is a time-frequency joint learning process implemented by a series of Transformer models. Experiments on IEMOCAP and CASIA databases indicate that our proposed method outdoes the state-of-the-art methods.