SeQuiFi: Mitigating Catastrophic Forgetting in Speech Emotion Recognition with Sequential Class-Finetuning

TL;DR

SeQuiFi introduces a sequential class-finetuning method that effectively reduces catastrophic forgetting in speech emotion recognition, outperforming existing techniques across diverse multilingual datasets.

Contribution

The paper presents SeQuiFi, a novel sequential class-finetuning approach that improves continual learning in speech emotion recognition by mitigating catastrophic forgetting.

Findings

SeQuiFi outperforms vanilla fine-tuning and SOTA methods in accuracy and F1 scores.

Effective across multiple multilingual speech emotion datasets.

Significantly reduces catastrophic forgetting in SER tasks.

Abstract

In this work, we introduce SeQuiFi, a novel approach for mitigating catastrophic forgetting (CF) in speech emotion recognition (SER). SeQuiFi adopts a sequential class-finetuning strategy, where the model is fine-tuned incrementally on one emotion class at a time, preserving and enhancing retention for each class. While various state-of-the-art (SOTA) methods, such as regularization-based, memory-based, and weight-averaging techniques, have been proposed to address CF, it still remains a challenge, particularly with diverse and multilingual datasets. Through extensive experiments, we demonstrate that SeQuiFi significantly outperforms both vanilla fine-tuning and SOTA continual learning techniques in terms of accuracy and F1 scores on multiple benchmark SER datasets, including CREMA-D, RAVDESS, Emo-DB, MESD, and SHEMO, covering different languages.