Speech Emotion: Investigating Model Representations, Multi-Task Learning and Knowledge Distillation

TL;DR

This paper explores using pre-trained model representations and knowledge distillation to enhance speech emotion recognition, achieving state-of-the-art results in valence estimation and emotion modeling across multiple datasets.

Contribution

It introduces a fusion approach of pre-trained embeddings and a distillation method to improve emotion recognition accuracy from speech signals.

Findings

Fusion of pre-trained embeddings improves valence CCC by 79%.

Knowledge distillation yields a 12% relative improvement.

Achieved new state-of-the-art results on MSP-Podcast datasets.

Abstract

Estimating dimensional emotions, such as activation, valence and dominance, from acoustic speech signals has been widely explored over the past few years. While accurate estimation of activation and dominance from speech seem to be possible, the same for valence remains challenging. Previous research has shown that the use of lexical information can improve valence estimation performance. Lexical information can be obtained from pre-trained acoustic models, where the learned representations can improve valence estimation from speech. We investigate the use of pre-trained model representations to improve valence estimation from acoustic speech signal. We also explore fusion of representations to improve emotion estimation across all three emotion dimensions: activation, valence and dominance. Additionally, we investigate if representations from pre-trained models can be distilled into models trained with low-level features, resulting in models with a less number of parameters. We show that fusion of pre-trained model embeddings result in a 79% relative improvement in concordance correlation coefficient CCC on valence estimation compared to standard acoustic feature baseline (mel-filterbank energies), while distillation from pre-trained model embeddings to lower-dimensional representations yielded a relative 12% improvement. Such performance gains were observed over two evaluation sets, indicating that our proposed architecture generalizes across those evaluation sets. We report new state-of-the-art "text-free" acoustic-only dimensional emotion estimation $CCC$ values on two MSP-Podcast evaluation sets.