EM2LDL: A Multilingual Speech Corpus for Mixed Emotion Recognition through Label Distribution Learning

TL;DR

EM2LDL is a new multilingual speech corpus that captures mixed emotions across English, Mandarin, and Cantonese, enabling better emotion recognition in diverse, real-world multilingual contexts.

Contribution

This paper introduces EM2LDL, a multilingual speech corpus with fine-grained emotion annotations, addressing the lack of diverse, ecological, and mixed emotion datasets for speech emotion recognition.

Findings

Self-supervised models perform robustly on EM2LDL

HuBERT-large-EN achieves the best results

Corpus enables exploration of complex emotional dynamics in multilingual speech

Abstract

This study introduces EM2LDL, a novel multilingual speech corpus designed to advance mixed emotion recognition through label distribution learning. Addressing the limitations of predominantly monolingual and single-label emotion corpora \textcolor{black}{that restrict linguistic diversity, are unable to model mixed emotions, and lack ecological validity}, EM2LDL comprises expressive utterances in English, Mandarin, and Cantonese, capturing the intra-utterance code-switching prevalent in multilingual regions like Hong Kong and Macao. The corpus integrates spontaneous emotional expressions from online platforms, annotated with fine-grained emotion distributions across 32 categories. Experimental baselines using self-supervised learning models demonstrate robust performance in speaker-independent gender-, age-, and personality-based evaluations, with HuBERT-large-EN achieving optimal results. By incorporating linguistic diversity and ecological validity, EM2LDL enables the exploration of complex emotional dynamics in multilingual settings. This work provides a versatile testbed for developing adaptive, empathetic systems for applications in affective computing, including mental health monitoring and cross-cultural communication. The dataset, annotations, and baseline codes are publicly available at https://github.com/xingfengli/EM2LDL.