Emotion-Aware Speech Self-Supervised Representation Learning with Intensity Knowledge

TL;DR

This paper introduces an emotion-aware speech SSL method that incorporates emotion intensity knowledge into the training process, improving emotion recognition performance.

Contribution

It proposes a novel emotional masking strategy using emotion intensities and applies it to Transformer and CNN-based models for better emotion representation.

Findings

Enhanced SER performance on IEMOCAP dataset

Emotion-aware masking improves speech emotion recognition

Applicable to Transformer and CNN models

Abstract

Speech Self-Supervised Learning (SSL) has demonstrated considerable efficacy in various downstream tasks. Nevertheless, prevailing self-supervised models often overlook the incorporation of emotion-related prior information, thereby neglecting the potential enhancement of emotion task comprehension through emotion prior knowledge in speech. In this paper, we propose an emotion-aware speech representation learning with intensity knowledge. Specifically, we extract frame-level emotion intensities using an established speech-emotion understanding model. Subsequently, we propose a novel emotional masking strategy (EMS) to incorporate emotion intensities into the masking process. We selected two representative models based on Transformer and CNN, namely MockingJay and Non-autoregressive Predictive Coding (NPC), and conducted experiments on IEMOCAP dataset. Experiments have demonstrated that the representations derived from our proposed method outperform the original model in SER task.