AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models

TL;DR

This paper introduces AffectGPT, a new multimodal large language model and a large-scale descriptive emotion dataset, MER-Caption, along with a benchmark, MER-UniBench, to advance emotion understanding in videos.

Contribution

The paper presents a novel dataset with over 115K samples and 2K emotion categories, a new model AffectGPT with enhanced multimodal integration, and a comprehensive benchmark for emotion understanding tasks.

Findings

AffectGPT achieves robust performance across MER tasks.

The MER-Caption dataset is the largest descriptive emotion dataset to date.

The MER-UniBench provides tailored evaluation metrics for MER.

Abstract

The emergence of multimodal large language models (MLLMs) advances multimodal emotion recognition (MER) to the next level, from naive discriminative tasks to complex emotion understanding with advanced video understanding abilities and natural language description. However, the current community suffers from a lack of large-scale datasets with intensive, descriptive emotion annotations, as well as a multimodal-centric framework to maximize the potential of MLLMs for emotion understanding. To address this, we establish a new benchmark for MLLM-based emotion understanding with a novel dataset (MER-Caption) and a new model (AffectGPT). Utilizing our model-based crowd-sourcing data collection strategy, we construct the largest descriptive emotion dataset to date (by far), featuring over 2K fine-grained emotion categories across 115K samples. We also introduce the AffectGPT model, designed with pre-fusion operations to enhance multimodal integration. Finally, we present MER-UniBench, a unified benchmark with evaluation metrics tailored for typical MER tasks and the free-form, natural language output style of MLLMs. Extensive experimental results show AffectGPT's robust performance across various MER tasks. We have released both the code and the dataset to advance research and development in emotion understanding: https://github.com/zeroQiaoba/AffectGPT.