Speech Emotion Recognition using Semantic Information

TL;DR

This paper introduces a novel speech emotion recognition framework that combines semantic and paralinguistic features using an attention mechanism and LSTM, achieving state-of-the-art results on the SEWA dataset.

Contribution

It proposes a new framework that integrates semantic and paralinguistic speech features with an attention mechanism for improved emotion recognition.

Findings

Achieves state-of-the-art results on valence and liking dimensions.

Effectively captures both semantic and paralinguistic information.

Outperforms previous top models on the SEWA dataset.

Abstract

Speech emotion recognition is a crucial problem manifesting in a multitude of applications such as human computer interaction and education. Although several advancements have been made in the recent years, especially with the advent of Deep Neural Networks (DNN), most of the studies in the literature fail to consider the semantic information in the speech signal. In this paper, we propose a novel framework that can capture both the semantic and the paralinguistic information in the signal. In particular, our framework is comprised of a semantic feature extractor, that captures the semantic information, and a paralinguistic feature extractor, that captures the paralinguistic information. Both semantic and paraliguistic features are then combined to a unified representation using a novel attention mechanism. The unified feature vector is passed through a LSTM to capture the temporal dynamics in the signal, before the final prediction. To validate the effectiveness of our framework, we use the popular SEWA dataset of the AVEC challenge series and compare with the three winning papers. Our model provides state-of-the-art results in the valence and liking dimensions.