Modulation spectral features for speech emotion recognition using deep neural networks

TL;DR

This paper introduces constant-Q transform based modulation spectral features (CQT-MSF) for speech emotion recognition, demonstrating improved performance over traditional features by capturing emotion-specific information through a biologically inspired spectral-temporal representation.

Contribution

The work proposes a novel CQT-MSF feature extraction method combined with deep learning, outperforming existing spectrogram and scattering transform features in speech emotion recognition.

Findings

CQT-MSF outperforms mel-scale spectrogram features on SER databases.

CQT-MSF surpasses scattering transform coefficients in emotion recognition.

Grad-CAM analysis confirms the importance of CQT-MSF features in SER.

Abstract

This work explores the use of constant-Q transform based modulation spectral features (CQT-MSF) for speech emotion recognition (SER). The human perception and analysis of sound comprise of two important cognitive parts: early auditory analysis and cortex-based processing. The early auditory analysis considers spectrogram-based representation whereas cortex-based analysis includes extraction of temporal modulations from the spectrogram. This temporal modulation representation of spectrogram is called modulation spectral feature (MSF). As the constant-Q transform (CQT) provides higher resolution at emotion salient low-frequency regions of speech, we find that CQT-based spectrogram, together with its temporal modulations, provides a representation enriched with emotion-specific information. We argue that CQT-MSF when used with a 2-dimensional convolutional network can provide a time-shift invariant and deformation insensitive representation for SER. Our results show that CQT-MSF outperforms standard mel-scale based spectrogram and its modulation features on two popular SER databases, Berlin EmoDB and RAVDESS. We also show that our proposed feature outperforms the shift and deformation invariant scattering transform coefficients, hence, showing the importance of joint hand-crafted and self-learned feature extraction instead of reliance on complete hand-crafted features. Finally, we perform Grad-CAM analysis to visually inspect the contribution of constant-Q modulation features over SER.