Ensembling Multilingual Pre-Trained Models for Predicting Multi-Label Regression Emotion Share from Speech

TL;DR

This paper introduces an ensemble approach combining multilingual pre-trained speech models to improve multi-label emotion share prediction, demonstrating enhanced performance over previous monolingual fusion methods.

Contribution

It proposes a novel ensemble learning method for multilingual speech emotion recognition, addressing data scarcity and cross-lingual evaluation challenges.

Findings

Ensemble learning improved Spearman correlation to 0.537 on test data.

Multilingual models outperform monolingual fusion methods.

The approach effectively handles emotion share regression from speech.

Abstract

Speech emotion recognition has evolved from research to practical applications. Previous studies of emotion recognition from speech have focused on developing models on certain datasets like IEMOCAP. The lack of data in the domain of emotion modeling emerges as a challenge to evaluate models in the other dataset, as well as to evaluate speech emotion recognition models that work in a multilingual setting. This paper proposes an ensemble learning to fuse results of pre-trained models for emotion share recognition from speech. The models were chosen to accommodate multilingual data from English and Spanish. The results show that ensemble learning can improve the performance of the baseline model with a single model and the previous best model from the late fusion. The performance is measured using the Spearman rank correlation coefficient since the task is a regression problem with ranking values. A Spearman rank correlation coefficient of 0.537 is reported for the test set, while for the development set, the score is 0.524. These scores are higher than the previous study of a fusion method from monolingual data, which achieved scores of 0.476 for the test and 0.470 for the development.