VidEmo: Affective-Tree Reasoning for Emotion-Centric Video Foundation Models

TL;DR

This paper introduces VidEmo, a novel emotion reasoning framework for videos that combines attribute perception, expression analysis, and emotional understanding, utilizing a new dataset and two-stage training to improve emotion analysis performance.

Contribution

The paper presents VidEmo, a new video emotion foundation model with a two-stage training process and a large emotion-centric dataset, advancing emotion reasoning in videos.

Findings

Achieves competitive performance across 15 face perception tasks.

Introduces Emo-CFG, a large dataset with 2.1 million samples.

Demonstrates effective emotion reasoning and instruction-following capabilities.

Abstract

Understanding and predicting emotion from videos has gathered significant attention in recent studies, driven by advancements in video large language models (VideoLLMs). While advanced methods have made progress in video emotion analysis, the intrinsic nature of emotions poses significant challenges. Emotions are characterized by dynamic and cues-dependent properties, making it difficult to understand complex and evolving emotional states with reasonable rationale. To tackle these challenges, we propose a novel affective cues-guided reasoning framework that unifies fundamental attribute perception, expression analysis, and high-level emotional understanding in a stage-wise manner. At the core of our approach is a family of video emotion foundation models (VidEmo), specifically designed for emotion reasoning and instruction-following. These models undergo a two-stage tuning process: first, curriculum emotion learning for injecting emotion knowledge, followed by affective-tree reinforcement learning for emotion reasoning. Moreover, we establish a foundational data infrastructure and introduce a emotion-centric fine-grained dataset (Emo-CFG) consisting of 2.1M diverse instruction-based samples. Emo-CFG includes explainable emotional question-answering, fine-grained captions, and associated rationales, providing essential resources for advancing emotion understanding tasks. Experimental results demonstrate that our approach achieves competitive performance, setting a new milestone across 15 face perception tasks.