MERBench: A Unified Evaluation Benchmark for Multimodal Emotion Recognition

TL;DR

MERBench provides a standardized evaluation framework for multimodal emotion recognition, enabling fair comparison of methods and guiding future research, while introducing a new Chinese emotion dataset MER2023 for diverse learning tasks.

Contribution

The paper introduces MERBench, a unified benchmark for fair comparison in multimodal emotion recognition, and presents MER2023, a new Chinese emotion dataset for advanced learning research.

Findings

MERBench enables consistent evaluation of multimodal emotion recognition methods.

Analysis reveals the impact of feature selection and fusion techniques.

MER2023 supports research on multi-label learning, noise robustness, and semi-supervised learning.

Abstract

Multimodal emotion recognition plays a crucial role in enhancing user experience in human-computer interaction. Over the past few decades, researchers have proposed a series of algorithms and achieved impressive progress. Although each method shows its superior performance, different methods lack a fair comparison due to inconsistencies in feature extractors, evaluation manners, and experimental settings. These inconsistencies severely hinder the development of this field. Therefore, we build MERBench, a unified evaluation benchmark for multimodal emotion recognition. We aim to reveal the contribution of some important techniques employed in previous works, such as feature selection, multimodal fusion, robustness analysis, fine-tuning, pre-training, etc. We hope this benchmark can provide clear and comprehensive guidance for follow-up researchers. Based on the evaluation results of MERBench, we further point out some promising research directions. Additionally, we introduce a new emotion dataset MER2023, focusing on the Chinese language environment. This dataset can serve as a benchmark dataset for research on multi-label learning, noise robustness, and semi-supervised learning. We encourage the follow-up researchers to evaluate their algorithms under the same experimental setup as MERBench for fair comparisons. Our code is available at: https://github.com/zeroQiaoba/MERTools.