Emo-bias: A Large Scale Evaluation of Social Bias on Speech Emotion Recognition

TL;DR

This study evaluates gender bias in Speech Emotion Recognition models trained with Self-Supervised Learning, revealing biases related to gender, language, and data distribution, with implications for fairer AI systems.

Contribution

It is the first large-scale analysis of gender bias in SSL-based SER models from both model and data perspectives.

Findings

Females show slightly higher SER performance than males.

Modified CPC and XLS-R models exhibit significant gender bias.

Bias is more influenced by training data distribution than upstream model representations.

Abstract

The rapid growth of Speech Emotion Recognition (SER) has diverse global applications, from improving human-computer interactions to aiding mental health diagnostics. However, SER models might contain social bias toward gender, leading to unfair outcomes. This study analyzes gender bias in SER models trained with Self-Supervised Learning (SSL) at scale, exploring factors influencing it. SSL-based SER models are chosen for their cutting-edge performance. Our research pioneering research gender bias in SER from both upstream model and data perspectives. Our findings reveal that females exhibit slightly higher overall SER performance than males. Modified CPC and XLS-R, two well-known SSL models, notably exhibit significant bias. Moreover, models trained with Mandarin datasets display a pronounced bias toward valence. Lastly, we find that gender-wise emotion distribution differences in training data significantly affect gender bias, while upstream model representation has a limited impact.