Pioneering Multimodal Emotion Recognition in the Era of Large Models: From Closed Sets to Open Vocabularies

TL;DR

This paper conducts the first large-scale benchmarking of multimodal large language models for open-vocabulary emotion recognition, analyzing their reasoning, fusion strategies, and modality importance to guide future emotion AI development.

Contribution

It introduces a comprehensive evaluation of 19 MLLMs on MER-OV, revealing optimal fusion strategies and the significance of video modality, and provides practical benchmarks and insights for advancing emotion recognition.

Findings

Two-stage trimodal fusion yields best performance.

Video modality is most critical for emotion recognition.

Narrow performance gap between open- and closed-source LLMs.

Abstract

Recent advances in multimodal large language models (MLLMs) have demonstrated remarkable multi- and cross-modal integration capabilities. However, their potential for fine-grained emotion understanding remains systematically underexplored. While open-vocabulary multimodal emotion recognition (MER-OV) has emerged as a promising direction to overcome the limitations of closed emotion sets, no comprehensive evaluation of MLLMs in this context currently exists. To address this, our work presents the first large-scale benchmarking study of MER-OV on the OV-MERD dataset, evaluating 19 mainstream MLLMs, including general-purpose, modality-specialized, and reasoning-enhanced architectures. Through systematic analysis of model reasoning capacity, fusion strategies, contextual utilization, and prompt design, we provide key insights into the capabilities and limitations of current MLLMs for MER-OV. Our evaluation reveals that a two-stage, trimodal (audio, video, and text) fusion approach achieves optimal performance in MER-OV, with video emerging as the most critical modality. We further identify a surprisingly narrow gap between open- and closed-source LLMs. These findings establish essential benchmarks and offer practical guidelines for advancing open-vocabulary and fine-grained affective computing, paving the way for more nuanced and interpretable emotion AI systems. Associated code will be made publicly available upon acceptance.