Multi-Loss Learning for Speech Emotion Recognition with Energy-Adaptive Mixup and Frame-Level Attention

TL;DR

This paper introduces a multi-loss learning framework for speech emotion recognition that combines energy-adaptive mixup, frame-level attention, and multiple loss functions to improve accuracy and robustness across several datasets.

Contribution

The paper proposes a novel multi-loss learning approach integrating energy-adaptive mixup and frame-level attention for enhanced speech emotion recognition performance.

Findings

Achieves state-of-the-art results on four SER datasets.

Effectively handles class imbalance and emotional variability.

Demonstrates robustness and improved feature discrimination.

Abstract

Speech emotion recognition (SER) is an important technology in human-computer interaction. However, achieving high performance is challenging due to emotional complexity and scarce annotated data. To tackle these challenges, we propose a multi-loss learning (MLL) framework integrating an energy-adaptive mixup (EAM) method and a frame-level attention module (FLAM). The EAM method leverages SNR-based augmentation to generate diverse speech samples capturing subtle emotional variations. FLAM enhances frame-level feature extraction for multi-frame emotional cues. Our MLL strategy combines Kullback-Leibler divergence, focal, center, and supervised contrastive loss to optimize learning, address class imbalance, and improve feature separability. We evaluate our method on four widely used SER datasets: IEMOCAP, MSP-IMPROV, RAVDESS, and SAVEE. The results demonstrate our method achieves state-of-the-art performance, suggesting its effectiveness and robustness.