Omni-Emotion: Extending Video MLLM with Detailed Face and Audio Modeling for Multimodal Emotion Analysis

TL;DR

This paper introduces Omni-Emotion, a multimodal emotion analysis framework that enhances video large language models by integrating detailed facial and audio cues, supported by new datasets, achieving state-of-the-art results.

Contribution

The paper presents a novel integration of facial encoding models into video MLLMs and introduces new datasets with detailed emotion annotations for improved multimodal emotion understanding.

Findings

Achieved state-of-the-art performance in emotion recognition and reasoning.

Developed new datasets with over 24,000 samples for training and 3,500 detailed annotations.

Effectively unified audio and facial cues within a shared feature space.

Abstract

Understanding emotions accurately is essential for fields like human-computer interaction. Due to the complexity of emotions and their multi-modal nature (e.g., emotions are influenced by facial expressions and audio), researchers have turned to using multi-modal models to understand human emotions rather than single-modality. However, current video multi-modal large language models (MLLMs) encounter difficulties in effectively integrating audio and identifying subtle facial micro-expressions. Furthermore, the lack of detailed emotion analysis datasets also limits the development of multimodal emotion analysis. To address these issues, we introduce a self-reviewed dataset and a human-reviewed dataset, comprising 24,137 coarse-grained samples and 3,500 manually annotated samples with detailed emotion annotations, respectively. These datasets allow models to learn from diverse scenarios and better generalize to real-world applications. Moreover, in addition to the audio modeling, we propose to explicitly integrate facial encoding models into the existing advanced Video MLLM, enabling the MLLM to effectively unify audio and the subtle facial cues for emotion understanding. By aligning these features within a unified space and employing instruction tuning in our proposed datasets, our Omni-Emotion achieves state-of-the-art performance in both emotion recognition and reasoning tasks.