Frame-level emotional state alignment method for speech emotion recognition

TL;DR

This paper introduces a novel frame-level emotional state alignment method for speech emotion recognition, improving accuracy by focusing on emotionally consistent frames using a fine-tuned HuBERT model with pseudo-labels.

Contribution

The paper proposes a new frame-level alignment approach with task-adaptive pretraining and clustering, enhancing speech emotion recognition performance over existing methods.

Findings

Outperforms state-of-the-art on IEMOCAP dataset

Effective use of pseudo-labels for frame-level emotion detection

Improved focus on emotionally relevant frames

Abstract

Speech emotion recognition (SER) systems aim to recognize human emotional state during human-computer interaction. Most existing SER systems are trained based on utterance-level labels. However, not all frames in an audio have affective states consistent with utterance-level label, which makes it difficult for the model to distinguish the true emotion of the audio and perform poorly. To address this problem, we propose a frame-level emotional state alignment method for SER. First, we fine-tune HuBERT model to obtain a SER system with task-adaptive pretraining (TAPT) method, and extract embeddings from its transformer layers to form frame-level pseudo-emotion labels with clustering. Then, the pseudo labels are used to pretrain HuBERT. Hence, the each frame output of HuBERT has corresponding emotional information. Finally, we fine-tune the above pretrained HuBERT for SER by adding an attention layer on the top of it, which can focus only on those frames that are emotionally more consistent with utterance-level label. The experimental results performed on IEMOCAP indicate that our proposed method performs better than state-of-the-art (SOTA) methods.