Exploring Multilingual Unseen Speaker Emotion Recognition: Leveraging Co-Attention Cues in Multitask Learning

TL;DR

This paper presents CAMuLeNet, a novel deep learning architecture that improves multilingual unseen speaker emotion recognition by using co-attention fusion and multitask learning, and benchmarks various pretrained encoders on multiple datasets.

Contribution

Introduction of CAMuLeNet architecture and benchmarking of pretrained encoders for multilingual unseen speaker emotion recognition.

Findings

CAMuLeNet achieves about 8% improvement over existing methods.

Benchmarking results show pretrained encoders vary in effectiveness.

A new Hindi SER dataset, BhavVani, is released.

Abstract

Advent of modern deep learning techniques has given rise to advancements in the field of Speech Emotion Recognition (SER). However, most systems prevalent in the field fail to generalize to speakers not seen during training. This study focuses on handling challenges of multilingual SER, specifically on unseen speakers. We introduce CAMuLeNet, a novel architecture leveraging co-attention based fusion and multitask learning to address this problem. Additionally, we benchmark pretrained encoders of Whisper, HuBERT, Wav2Vec2.0, and WavLM using 10-fold leave-speaker-out cross-validation on five existing multilingual benchmark datasets: IEMOCAP, RAVDESS, CREMA-D, EmoDB and CaFE and, release a novel dataset for SER on the Hindi language (BhavVani). CAMuLeNet shows an average improvement of approximately 8% over all benchmarks on unseen speakers determined by our cross-validation strategy.