Improved Speech Emotion Recognition using Transfer Learning and Spectrogram Augmentation

TL;DR

This paper enhances speech emotion recognition by combining transfer learning from speaker recognition models with spectrogram augmentation, addressing data scarcity and improving accuracy on the IEMOCAP dataset.

Contribution

It introduces a transfer learning approach using a pre-trained ResNet with statistics pooling and applies spectrogram augmentation to boost SER performance.

Findings

Transfer learning improves emotion classification accuracy.

Spectrogram augmentation enhances model generalization.

Combined methods achieve state-of-the-art results on IEMOCAP.

Abstract

Automatic speech emotion recognition (SER) is a challenging task that plays a crucial role in natural human-computer interaction. One of the main challenges in SER is data scarcity, i.e., insufficient amounts of carefully labeled data to build and fully explore complex deep learning models for emotion classification. This paper aims to address this challenge using a transfer learning strategy combined with spectrogram augmentation. Specifically, we propose a transfer learning approach that leverages a pre-trained residual network (ResNet) model including a statistics pooling layer from speaker recognition trained using large amounts of speaker-labeled data. The statistics pooling layer enables the model to efficiently process variable-length input, thereby eliminating the need for sequence truncation which is commonly used in SER systems. In addition, we adopt a spectrogram augmentation technique to generate additional training data samples by applying random time-frequency masks to log-mel spectrograms to mitigate overfitting and improve the generalization of emotion recognition models. We evaluate the effectiveness of our proposed approach on the interactive emotional dyadic motion capture (IEMOCAP) dataset. Experimental results indicate that the transfer learning and spectrogram augmentation approaches improve the SER performance, and when combined achieve state-of-the-art results.