Video Affective Effects Prediction with Multi-modal Fusion and Shot-Long Temporal Context

TL;DR

This paper introduces a novel multi-modal fusion framework with shot-long temporal context modeling for predicting emotional impact in videos, significantly improving accuracy over existing methods.

Contribution

The paper proposes a comprehensive framework with modality-specific feature extraction, two-scale temporal structures, and a residual-based progressive fusion strategy for emotion prediction.

Findings

Achieved superior performance on the LIRIS-ACCEDE dataset.

Effectively models intra- and inter-clip temporal dependencies.

Enhances multi-modal fusion with residual-based training.

Abstract

Predicting the emotional impact of videos using machine learning is a challenging task considering the varieties of modalities, the complicated temporal contex of the video as well as the time dependency of the emotional states. Feature extraction, multi-modal fusion and temporal context fusion are crucial stages for predicting valence and arousal values in the emotional impact, but have not been successfully exploited. In this paper, we propose a comprehensive framework with novel designs of modal structure and multi-modal fusion strategy. We select the most suitable modalities for valence and arousal tasks respectively and each modal feature is extracted using the modality-specific pre-trained deep model on large generic dataset. Two-time-scale structures, one for the intra-clip and the other for the inter-clip, are proposed to capture the temporal dependency of video content and emotion states. To combine the complementary information from multiple modalities, an effective and efficient residual-based progressive training strategy is proposed. Each modality is step-wisely combined into the multi-modal model, responsible for completing the missing parts of features. With all those improvements above, our proposed prediction framework achieves better performance on the LIRIS-ACCEDE dataset with a large margin compared to the state-of-the-art.