"How to Explore Biases in Speech Emotion AI with Users?" A Speech-Emotion-Acting Study Exploring Age and Language Biases

TL;DR

This study investigates how age and language influence deliberate vocal emotion expression and how current speech emotion recognition systems interpret these variations, revealing robustness across groups but also highlighting limitations and biases.

Contribution

It introduces a novel experimental paradigm to assess biases in speech emotion AI across age and language, emphasizing human-centered evaluation methods.

Findings

No significant differences in model interpretation across age and language groups.

Models show robustness but have limitations in high-arousal emotion recognition.

Highlights the importance of human-centered metrics over system-centered accuracy.

Abstract

This study explores how age and language shape the deliberate vocal expression of emotion, addressing underexplored user groups, Teenagers (N = 12) and Adults 55+ (N = 12), within speech emotion recognition (SER). While most SER systems are trained on spontaneous, monolingual English data, our research evaluates how such models interpret intentionally performed emotional speech across age groups and languages (Danish and English). To support this, we developed a novel experimental paradigm combining a custom user interface with a backend for real-time SER prediction and data logging. Participants were prompted to hit visual targets in valence-arousal space by deliberately expressing four emotion targets. While limitations include some reliance on self-managed voice recordings and inconsistent task execution, the results suggest contrary to expectations, no significant differences between language or age groups, and a degree of cross-linguistic and age robustness in model interpretation. Though some limitations in high-arousal emotion recognition were evident. Our qualitative findings highlight the need to move beyond system-centered accuracy metrics and embrace more inclusive, human-centered SER models. By framing emotional expression as a goal-directed act and logging the real-time gap between human intent and machine interpretation, we expose the risks of affective misalignment.