Fast Yet Effective Speech Emotion Recognition with Self-distillation

TL;DR

This paper introduces a self-distillation approach for speech emotion recognition that enhances accuracy and efficiency, especially on resource-limited devices, by fine-tuning a pretrained model and its shallower versions simultaneously.

Contribution

It proposes a novel self-distillation framework for SER that improves performance, reduces inference time, and enables adaptive accuracy-efficiency trade-offs on edge devices.

Findings

Outperforms existing models on SER datasets.

Enables resource-efficient inference with shallower models.

Achieves state-of-the-art accuracy with less labeled data.

Abstract

Speech emotion recognition (SER) is the task of recognising human's emotional states from speech. SER is extremely prevalent in helping dialogue systems to truly understand our emotions and become a trustworthy human conversational partner. Due to the lengthy nature of speech, SER also suffers from the lack of abundant labelled data for powerful models like deep neural networks. Pre-trained complex models on large-scale speech datasets have been successfully applied to SER via transfer learning. However, fine-tuning complex models still requires large memory space and results in low inference efficiency. In this paper, we argue achieving a fast yet effective SER is possible with self-distillation, a method of simultaneously fine-tuning a pretrained model and training shallower versions of itself. The benefits of our self-distillation framework are threefold: (1) the adoption of self-distillation method upon the acoustic modality breaks through the limited ground-truth of speech data, and outperforms the existing models' performance on an SER dataset; (2) executing powerful models at different depth can achieve adaptive accuracy-efficiency trade-offs on resource-limited edge devices; (3) a new fine-tuning process rather than training from scratch for self-distillation leads to faster learning time and the state-of-the-art accuracy on data with small quantities of label information.