Dynamic Layer Customization for Noise Robust Speech Emotion Recognition in Heterogeneous Condition Training

TL;DR

This paper proposes a dynamic layer customization approach for noise-robust speech emotion recognition, improving performance in heterogeneous noise conditions through specialized feature encoding and domain adaptation, applicable to unimodal and multimodal systems.

Contribution

It introduces a novel method for dynamic routing of samples to specialized encoders based on noise conditions, enhancing robustness without relying on unlabelled test data.

Findings

Significant performance improvements with full knowledge of test noise conditions.

Effective generalization to unseen noise conditions using domain adaptation.

Enhanced multimodal emotion recognition by maintaining temporal order.

Abstract

Robustness to environmental noise is important to creating automatic speech emotion recognition systems that are deployable in the real world. Prior work on noise robustness has assumed that systems would not make use of sample-by-sample training noise conditions, or that they would have access to unlabelled testing data to generalize across noise conditions. We avoid these assumptions and introduce the resulting task as heterogeneous condition training. We show that with full knowledge of the test noise conditions, we can improve performance by dynamically routing samples to specialized feature encoders for each noise condition, and with partial knowledge, we can use known noise conditions and domain adaptation algorithms to train systems that generalize well to unseen noise conditions. We then extend these improvements to the multimodal setting by dynamically routing samples to maintain temporal ordering, resulting in significant improvements over approaches that do not specialize or generalize based on noise type.