Two-stage dimensional emotion recognition by fusing predictions of acoustic and text networks using SVM

TL;DR

This paper presents a two-stage late-fusion method combining acoustic and text features for dimensional speech emotion recognition, achieving higher accuracy than previous approaches by using deep learning and SVM.

Contribution

It introduces a novel two-stage fusion framework that separately trains acoustic and text models and then combines their predictions with SVM for improved emotion recognition.

Findings

Outperforms single-modality models in emotion prediction accuracy.

Achieves higher concordance correlation coefficients than early fusion methods.

Demonstrates the effectiveness of late fusion in dimensional emotion modeling.

Abstract

Automatic speech emotion recognition (SER) by a computer is a critical component for more natural human-machine interaction. As in human-human interaction, the capability to perceive emotion correctly is essential to take further steps in a particular situation. One issue in SER is whether it is necessary to combine acoustic features with other data such as facial expressions, text, and motion capture. This research proposes to combine acoustic and text information by applying a late-fusion approach consisting of two steps. First, acoustic and text features are trained separately in deep learning systems. Second, the prediction results from the deep learning systems are fed into a support vector machine (SVM) to predict the final regression score. Furthermore, the task in this research is dimensional emotion modeling because it can enable a deeper analysis of affective states. Experimental results show that this two-stage, late-fusion approach, obtains higher performance than that of any one-stage processing, with a linear correlation from one-stage to two-stage processing. This late-fusion approach improves previous early fusion results measured in concordance correlation coefficients score.