MEmoBERT: Pre-training Model with Prompt-based Learning for Multimodal Emotion Recognition

TL;DR

MEmoBERT is a pre-training model utilizing prompt-based learning and self-supervised training on large-scale unlabeled videos to improve multimodal emotion recognition accuracy.

Contribution

It introduces a novel prompt-based pre-training approach that aligns the downstream emotion classification task with the pre-training task, enhancing performance.

Findings

Significant performance improvements on IEMOCAP and MSP-IMPROV datasets.

Effective learning of multimodal joint representations from unlabeled data.

Demonstrates the benefit of prompt-based reformulation in emotion recognition.

Abstract

Multimodal emotion recognition study is hindered by the lack of labelled corpora in terms of scale and diversity, due to the high annotation cost and label ambiguity. In this paper, we propose a pre-training model \textbf{MEmoBERT} for multimodal emotion recognition, which learns multimodal joint representations through self-supervised learning from large-scale unlabeled video data that come in sheer volume. Furthermore, unlike the conventional "pre-train, finetune" paradigm, we propose a prompt-based method that reformulates the downstream emotion classification task as a masked text prediction one, bringing the downstream task closer to the pre-training. Extensive experiments on two benchmark datasets, IEMOCAP and MSP-IMPROV, show that our proposed MEmoBERT significantly enhances emotion recognition performance.