Semi-supervised Multi-modal Emotion Recognition with Cross-Modal Distribution Matching

TL;DR

This paper introduces a semi-supervised multi-modal emotion recognition model that uses cross-modal distribution matching to effectively leverage unlabeled data, improving performance over existing methods on benchmark datasets.

Contribution

The paper proposes a novel semi-supervised approach based on cross-modality distribution matching for multi-modal emotion recognition, addressing data scarcity and label ambiguity issues.

Findings

Outperforms state-of-the-art methods on IEMOCAP and MELD datasets.

Effectively utilizes unlabeled data to enhance emotion recognition.

Achieves competitive results without relying on auxiliary information.

Abstract

Automatic emotion recognition is an active research topic with wide range of applications. Due to the high manual annotation cost and inevitable label ambiguity, the development of emotion recognition dataset is limited in both scale and quality. Therefore, one of the key challenges is how to build effective models with limited data resource. Previous works have explored different approaches to tackle this challenge including data enhancement, transfer learning, and semi-supervised learning etc. However, the weakness of these existing approaches includes such as training instability, large performance loss during transfer, or marginal improvement. In this work, we propose a novel semi-supervised multi-modal emotion recognition model based on cross-modality distribution matching, which leverages abundant unlabeled data to enhance the model training under the assumption that the inner emotional status is consistent at the utterance level across modalities. We conduct extensive experiments to evaluate the proposed model on two benchmark datasets, IEMOCAP and MELD. The experiment results prove that the proposed semi-supervised learning model can effectively utilize unlabeled data and combine multi-modalities to boost the emotion recognition performance, which outperforms other state-of-the-art approaches under the same condition. The proposed model also achieves competitive capacity compared with existing approaches which take advantage of additional auxiliary information such as speaker and interaction context.