Are Paralinguistic Representations all that is needed for Speech Emotion Recognition?

TL;DR

This paper evaluates the effectiveness of paralinguistic pre-trained model representations for speech emotion recognition across multiple languages and benchmarks, highlighting TRILLsson's superior performance.

Contribution

It provides a comprehensive comparison of five state-of-the-art PTM representations for SER in multilingual settings and benchmarks, revealing TRILLsson's advantages.

Findings

TRILLsson outperforms other PTMs in SER tasks.

Paralinguistic PTMs effectively capture pitch and tone.

Performance varies across languages and benchmarks.

Abstract

Availability of representations from pre-trained models (PTMs) have facilitated substantial progress in speech emotion recognition (SER). Particularly, representations from PTM trained for paralinguistic speech processing have shown state-of-the-art (SOTA) performance for SER. However, such paralinguistic PTM representations haven't been evaluated for SER in linguistic environments other than English. Also, paralinguistic PTM representations haven't been investigated in benchmarks such as SUPERB, EMO-SUPERB, ML-SUPERB for SER. This makes it difficult to access the efficacy of paralinguistic PTM representations for SER in multiple languages. To fill this gap, we perform a comprehensive comparative study of five SOTA PTM representations. Our results shows that paralinguistic PTM (TRILLsson) representations performs the best and this performance can be attributed to its effectiveness in capturing pitch, tone and other speech characteristics more effectively than other PTM representations.