MFSN: Multi-perspective Fusion Search Network For Pre-training Knowledge in Speech Emotion Recognition

TL;DR

This paper introduces MFSN, a novel pre-training framework for Speech Emotion Recognition that leverages multi-perspective fusion of semantic and acoustic cues, improving emotional understanding accuracy.

Contribution

The paper proposes a new architecture search space and a dual-perspective approach to comprehensively and appropriately capture emotional cues in speech.

Findings

MFSN outperforms existing methods on multiple datasets.

The dual-perspective approach improves emotion recognition accuracy.

The architecture search space effectively leverages semantic and acoustic cues.

Abstract

Speech Emotion Recognition (SER) is an important research topic in human-computer interaction. Many recent works focus on directly extracting emotional cues through pre-trained knowledge, frequently overlooking considerations of appropriateness and comprehensiveness. Therefore, we propose a novel framework for pre-training knowledge in SER, called Multi-perspective Fusion Search Network (MFSN). Considering comprehensiveness, we partition speech knowledge into Textual-related Emotional Content (TEC) and Speech-related Emotional Content (SEC), capturing cues from both semantic and acoustic perspectives, and we design a new architecture search space to fully leverage them. Considering appropriateness, we verify the efficacy of different modeling approaches in capturing SEC and fills the gap in current research. Experimental results on multiple datasets demonstrate the superiority of MFSN.