A Change of Heart: Improving Speech Emotion Recognition through Speech-to-Text Modality Conversion

TL;DR

This paper explores converting speech to text to improve emotion recognition accuracy, demonstrating that text-based methods outperform traditional speech-based approaches on the MELD dataset.

Contribution

Introduces a novel modality conversion approach for speech emotion recognition, showing significant improvements over existing speech-based methods.

Findings

Modality conversion with ASR improves SER performance.

Text-based classification outperforms speech-based methods.

Method achieves state-of-the-art results on MELD dataset.

Abstract

Speech Emotion Recognition (SER) is a challenging task. In this paper, we introduce a modality conversion concept aimed at enhancing emotion recognition performance on the MELD dataset. We assess our approach through two experiments: first, a method named Modality-Conversion that employs automatic speech recognition (ASR) systems, followed by a text classifier; second, we assume perfect ASR output and investigate the impact of modality conversion on SER, this method is called Modality-Conversion++. Our findings indicate that the first method yields substantial results, while the second method outperforms state-of-the-art (SOTA) speech-based approaches in terms of SER weighted-F1 (WF1) score on the MELD dataset. This research highlights the potential of modality conversion for tasks that can be conducted in alternative modalities.