Enhancing Segment-Based Speech Emotion Recognition by Deep Self-Learning

TL;DR

This paper introduces a deep self-learning framework for segment-based speech emotion recognition that iteratively refines noisy segment labels, significantly improving model performance on emotional speech datasets.

Contribution

It proposes a novel deep self-learning approach that dynamically corrects segment labels, addressing label noise issues in segment-based speech emotion recognition.

Findings

Significant performance improvements on three emotional corpora.

Effective label correction reduces noise impact.

Enhanced robustness of emotion recognition models.

Abstract

Despite the widespread utilization of deep neural networks (DNNs) for speech emotion recognition (SER), they are severely restricted due to the paucity of labeled data for training. Recently, segment-based approaches for SER have been evolving, which train backbone networks on shorter segments instead of whole utterances, and thus naturally augments training examples without additional resources. However, one core challenge remains for segment-based approaches: most emotional corpora do not provide ground-truth labels at the segment level. To supervisely train a segment-based emotion model on such datasets, the most common way assigns each segment the corresponding utterance's emotion label. However, this practice typically introduces noisy (incorrect) labels as emotional information is not uniformly distributed across the whole utterance. On the other hand, DNNs have been shown to easily over-fit a dataset when being trained with noisy labels. To this end, this work proposes a simple and effective deep self-learning (DSL) framework, which comprises a procedure to progressively correct segment-level labels in an iterative learning manner. The DSL method produces dynamically-generated and soft emotion labels, leading to significant performance improvements. Experiments on three well-known emotional corpora demonstrate noticeable gains using the proposed method.