VCEMO: Multi-Modal Emotion Recognition for Chinese Voiceprints

TL;DR

This paper introduces the VCEMO dataset for Chinese voiceprint emotion recognition and proposes a multimodal model that fuses speech, text, and external knowledge, achieving state-of-the-art results.

Contribution

It provides a new high-quality Chinese voiceprint emotion dataset and a multimodal fusion model with contrastive learning for improved emotion recognition.

Findings

Significant improvement over SOTA on VCEMO and IEMOCAP datasets

Effective fusion of speech, text, and external knowledge

Addresses dataset scarcity for Chinese voiceprint emotion recognition

Abstract

Emotion recognition can enhance humanized machine responses to user commands, while voiceprint-based perception systems can be easily integrated into commonly used devices like smartphones and stereos. Despite having the largest number of speakers, there is a noticeable absence of high-quality corpus datasets for emotion recognition using Chinese voiceprints. Hence, this paper introduces the VCEMO dataset to address this deficiency. The proposed dataset is constructed from everyday conversations and comprises over 100 users and 7,747 textual samples. Furthermore, this paper proposes a multimodal-based model as a benchmark, which effectively fuses speech, text, and external knowledge using a co-attention structure. The system employs contrastive learning-based regulation for the uneven distribution of the dataset and the diversity of emotional expressions. The experiments demonstrate the significant improvement of the proposed model over SOTA on the VCEMO and IEMOCAP datasets. Code and dataset will be released for research.