Multimodal Speech Emotion Recognition Using Modality-specific Self-Supervised Frameworks

TL;DR

This paper introduces a modality-specific self-supervised transformer framework for speech and text emotion recognition, leveraging data-efficient learning and multimodal fusion to outperform existing methods on the IEMOCAP dataset.

Contribution

It presents a novel self-supervised learning approach using transformer models for emotion recognition from speech and text, with effective multimodal fusion.

Findings

Achieved 77.58% accuracy on IEMOCAP dataset

Outperformed state-of-the-art emotion recognition methods

Demonstrated effectiveness of modality-specific pre-trained transformers

Abstract

Emotion recognition is a topic of significant interest in assistive robotics due to the need to equip robots with the ability to comprehend human behavior, facilitating their effective interaction in our society. Consequently, efficient and dependable emotion recognition systems supporting optimal human-machine communication are required. Multi-modality (including speech, audio, text, images, and videos) is typically exploited in emotion recognition tasks. Much relevant research is based on merging multiple data modalities and training deep learning models utilizing low-level data representations. However, most existing emotion databases are not large (or complex) enough to allow machine learning approaches to learn detailed representations. This paper explores modalityspecific pre-trained transformer frameworks for self-supervised learning of speech and text representations for data-efficient emotion recognition while achieving state-of-the-art performance in recognizing emotions. This model applies feature-level fusion using nonverbal cue data points from motion capture to provide multimodal speech emotion recognition. The model was trained using the publicly available IEMOCAP dataset, achieving an overall accuracy of 77.58% for four emotions, outperforming state-of-the-art approaches