Beyond Emotion Recognition: A Multi-Turn Multimodal Emotion Understanding and Reasoning Benchmark

TL;DR

This paper introduces a comprehensive benchmark for multi-turn multimodal emotion understanding and reasoning, highlighting the challenges faced by current models and proposing a multi-agent framework to enhance reasoning capabilities.

Contribution

The paper presents a new benchmark with real-life video data and questions for emotion reasoning, and proposes a multi-agent system to improve multimodal emotion understanding.

Findings

Existing models struggle with emotion reasoning tasks.

The multi-agent framework shows potential in addressing reasoning challenges.

Benchmark data covers diverse real-life scenarios and questions.

Abstract

Multimodal large language models (MLLMs) have been widely applied across various fields due to their powerful perceptual and reasoning capabilities. In the realm of psychology, these models hold promise for a deeper understanding of human emotions and behaviors. However, recent research primarily focuses on enhancing their emotion recognition abilities, leaving the substantial potential in emotion reasoning, which is crucial for improving the naturalness and effectiveness of human-machine interactions. Therefore, in this paper, we introduce a multi-turn multimodal emotion understanding and reasoning (MTMEUR) benchmark, which encompasses 1,451 video data from real-life scenarios, along with 5,101 progressive questions. These questions cover various aspects, including emotion recognition, potential causes of emotions, future action prediction, etc. Besides, we propose a multi-agent framework, where each agent specializes in a specific aspect, such as background context, character dynamics, and event details, to improve the system's reasoning capabilities. Furthermore, we conduct experiments with existing MLLMs and our agent-based method on the proposed benchmark, revealing that most models face significant challenges with this task.