Enhancing Video Memorability Prediction with Text-Motion Cross-modal Contrastive Loss and Its Application in Video Summarization

TL;DR

This paper introduces a novel cross-modal contrastive loss to improve motion feature representation in video memorability prediction, achieving state-of-the-art results and applying it to enhance video summarization.

Contribution

The paper proposes the Text-Motion Cross-modal Contrastive Loss (TMCCL) to better utilize motion cues and introduces MWCVS to improve video summarization using memorability prediction.

Findings

Achieved state-of-the-art performance on two datasets.

Demonstrated effectiveness of memorability in video summarization.

Enhanced motion feature representation through TMCCL.

Abstract

Video memorability refers to the ability of videos to be recalled after viewing, playing a crucial role in creating content that remains memorable. Existing models typically focus on extracting multimodal features to predict video memorability scores but often fail to fully utilize motion cues. The representation of motion features is compromised during the fine-tuning phase of the motion feature extractor due to a lack of labeled data. In this paper, we introduce the Text-Motion Cross-modal Contrastive Loss (TMCCL), a multimodal video memorability prediction model designed to enhance the representation of motion features. We tackle the challenge of improving motion feature representation by leveraging text description similarities across videos to establish positive and negative motion sample sets for a given target. This enhancement allows the model to learn similar feature representations for semantically related motion content, resulting in more accurate memorability predictions. Our model achieves state-of-the-art performance on two video memorability prediction datasets. Moreover, the potential applications of video memorability prediction have been underexplored. To address this gap, we present Memorability Weighted Correction for Video Summarization (MWCVS), using video memorability prediction to reduce subjectivity in video summarization labels. Experimental results on two video summarization datasets demonstrate the effectiveness of MWCVS, showcasing the promising applications of video memorability prediction.