Stimulus Modality Matters: Impact of Perceptual Evaluations from Different Modalities on Speech Emotion Recognition System Performance

TL;DR

This paper investigates how the modality of perceptual evaluations used for labeling affects the performance of speech emotion recognition systems, finding voice-only labels yield better results.

Contribution

It provides a comprehensive comparison of emotion labels elicited from different modalities and introduces an all-inclusive label approach for training SER systems.

Findings

Voice-only elicited labels improve SER performance.

Multimodal labels do not outperform voice-only labels.

An all-inclusive label combining multiple modalities was evaluated.

Abstract

Speech Emotion Recognition (SER) systems rely on speech input and emotional labels annotated by humans. However, various emotion databases collect perceptional evaluations in different ways. For instance, the IEMOCAP dataset uses video clips with sounds for annotators to provide their emotional perceptions. However, the most significant English emotion dataset, the MSP-PODCAST, only provides speech for raters to choose the emotional ratings. Nevertheless, using speech as input is the standard approach to training SER systems. Therefore, the open question is the emotional labels elicited by which scenarios are the most effective for training SER systems. We comprehensively compare the effectiveness of SER systems trained with labels elicited by different modality stimuli and evaluate the SER systems on various testing conditions. Also, we introduce an all-inclusive label that combines all labels elicited by various modalities. We show that using labels elicited by voice-only stimuli for training yields better performance on the test set, whereas labels elicited by voice-only stimuli.