AffectGPT: Dataset and Framework for Explainable Multimodal Emotion Recognition

TL;DR

AffectGPT introduces a large-scale, coarsely-labeled multimodal emotion dataset and a two-stage training framework to improve explainable emotion recognition, reducing annotation costs and enhancing model performance.

Contribution

The paper presents EMER-Coarse dataset construction and a novel two-stage AffectGPT training framework for better multimodal emotion recognition.

Findings

AffectGPT outperforms baseline models on EMER tasks.

The EMER-Coarse dataset significantly expands available data.

Two-stage training improves alignment with manual annotations.

Abstract

Explainable Multimodal Emotion Recognition (EMER) is an emerging task that aims to achieve reliable and accurate emotion recognition. However, due to the high annotation cost, the existing dataset (denoted as EMER-Fine) is small, making it difficult to perform supervised training. To reduce the annotation cost and expand the dataset size, this paper reviews the previous dataset construction process. Then, we simplify the annotation pipeline, avoid manual checks, and replace the closed-source models with open-source models. Finally, we build \textbf{EMER-Coarse}, a coarsely-labeled dataset containing large-scale samples. Besides the dataset, we propose a two-stage training framework \textbf{AffectGPT}. The first stage exploits EMER-Coarse to learn a coarse mapping between multimodal inputs and emotion-related descriptions; the second stage uses EMER-Fine to better align with manually-checked results. Experimental results demonstrate the effectiveness of our proposed method on the challenging EMER task. To facilitate further research, we will make the code and dataset available at: https://github.com/zeroQiaoba/AffectGPT.