Metadata-Enhanced Speech Emotion Recognition: Augmented Residual Integration and Co-Attention in Two-Stage Fine-Tuning

TL;DR

This paper introduces a novel two-stage fine-tuning approach for Speech Emotion Recognition that leverages metadata through Augmented Residual Integration and Co-attention modules, significantly improving performance over state-of-the-art models.

Contribution

The paper proposes a new two-stage fine-tuning method with ARI and Co-attention modules to effectively utilize metadata in SSL models for SER.

Findings

Outperforms SOTA models on IEMOCAP dataset

Enhances auxiliary task performance with ARI module

Effectively utilizes multidimensional metadata information

Abstract

Speech Emotion Recognition (SER) involves analyzing vocal expressions to determine the emotional state of speakers, where the comprehensive and thorough utilization of audio information is paramount. Therefore, we propose a novel approach on self-supervised learning (SSL) models that employs all available auxiliary information -- specifically metadata -- to enhance performance. Through a two-stage fine-tuning method in multi-task learning, we introduce the Augmented Residual Integration (ARI) module, which enhances transformer layers in encoder of SSL models. The module efficiently preserves acoustic features across all different levels, thereby significantly improving the performance of metadata-related auxiliary tasks that require various levels of features. Moreover, the Co-attention module is incorporated due to its complementary nature with ARI, enabling the model to effectively utilize multidimensional information and contextual relationships from metadata-related auxiliary tasks. Under pre-trained base models and speaker-independent setup, our approach consistently surpasses state-of-the-art (SOTA) models on multiple SSL encoders for the IEMOCAP dataset.