Multi-Channel Speech Enhancement for Cocktail Party Speech Emotion Recognition

TL;DR

This paper demonstrates that multi-channel speech enhancement significantly improves emotion recognition accuracy in cocktail party scenarios by effectively separating speech from noise and interference, outperforming single-channel methods in various datasets.

Contribution

The study introduces a novel multi-channel speech dereverberation and separation front-end integrating DNN-WPE and mask-based MVDR, enhancing emotion recognition in noisy environments.

Findings

MCSE outperforms single-channel baselines in accuracy and F1 scores.

Significant improvements of up to 9.5% in weighted accuracy.

Effective zero-shot generalization to out-of-domain data.

Abstract

This paper highlights the critical importance of multi-channel speech enhancement (MCSE) for speech emotion recognition (ER) in cocktail party scenarios. A multi-channel speech dereverberation and separation front-end integrating DNN-WPE and mask-based MVDR is used to extract the target speaker's speech from the mixture speech, before being fed into the downstream ER back-end using HuBERT- and ViT-based speech and visual features. Experiments on mixture speech constructed using the IEMOCAP and MSP-FACE datasets suggest the MCSE output consistently outperforms domain fine-tuned single-channel speech representations produced by: a) Conformer-based metric GANs; and b) WavLM SSL features with optional SE-ER dual task fine-tuning. Statistically significant increases in weighted, unweighted accuracy and F1 measures by up to 9.5%, 8.5% and 9.1% absolute (17.1%, 14.7% and 16.0% relative) are obtained over the above single-channel baselines. The generalization of IEMOCAP trained MCSE front-ends are also shown when being zero-shot applied to out-of-domain MSP-FACE data.