Charting 15 years of progress in deep learning for speech emotion recognition: A replication study

TL;DR

This study reviews 15 years of deep learning advancements in speech emotion recognition, revealing diminishing returns with newer models and emphasizing the importance of model selection in perceived progress.

Contribution

It provides a comprehensive quantification of progress in SER over 15 years and analyzes the impact of different model architectures, highlighting the plateau after transformer models.

Findings

Diminishing returns observed with recent deep learning models

Plateau in progress after transformer architectures

Perceptions of progress depend on model comparison choices

Abstract

Speech emotion recognition (SER) has long benefited from the adoption of deep learning methodologies. Deeper models -- with more layers and more trainable parameters -- are generally perceived as being `better' by the SER community. This raises the question -- \emph{how much better} are modern-era deep neural networks compared to their earlier iterations? Beyond that, the more important question of how to move forward remains as poignant as ever. SER is far from a solved problem; therefore, identifying the most prominent avenues of future research is of paramount importance. In the present contribution, we attempt a quantification of progress in the 15 years of research beginning with the introduction of the landmark 2009 INTERSPEECH Emotion Challenge. We conduct a large scale investigation of model architectures, spanning both audio-based models that rely on speech inputs and text-baed models that rely solely on transcriptions. Our results point towards diminishing returns and a plateau after the recent introduction of transformer architectures. Moreover, we demonstrate how perceptions of progress are conditioned on the particular selection of models that are compared. Our findings have important repercussions about the state-of-the-art in SER research and the paths forward