Multilingual Speech Emotion Recognition With Multi-Gating Mechanism and Neural Architecture Search

TL;DR

This paper introduces a multilingual speech emotion recognition model that leverages multi-gating and neural architecture search to improve accuracy across languages, especially low-resource ones.

Contribution

It proposes a novel multi-domain, language-specific SER model with multi-gating and neural architecture search, enhancing performance on low-resource languages.

Findings

Achieved 3% accuracy improvement for German

Achieved 14.3% accuracy improvement for French

Introduced contrastive auxiliary loss for better feature separation

Abstract

Speech emotion recognition (SER) classifies audio into emotion categories such as Happy, Angry, Fear, Disgust and Neutral. While Speech Emotion Recognition (SER) is a common application for popular languages, it continues to be a problem for low-resourced languages, i.e., languages with no pretrained speech-to-text recognition models. This paper firstly proposes a language-specific model that extract emotional information from multiple pre-trained speech models, and then designs a multi-domain model that simultaneously performs SER for various languages. Our multidomain model employs a multi-gating mechanism to generate unique weighted feature combination for each language, and also searches for specific neural network structure for each language through a neural architecture search module. In addition, we introduce a contrastive auxiliary loss to build more separable representations for audio data. Our experiments show that our model raises the state-of-the-art accuracy by 3% for German and 14.3% for French.