MF2Summ: Multimodal Fusion for Video Summarization with Temporal Alignment

TL;DR

MF2Summ is a multimodal video summarization model that combines visual and auditory data using advanced attention mechanisms to produce more comprehensive video summaries, showing improved performance over existing methods.

Contribution

This paper introduces MF2Summ, a novel multimodal fusion approach with temporal alignment for video summarization, utilizing cross-modal Transformers and alignment-guided self-attention.

Findings

Achieves higher F1-scores on SumMe and TVSum datasets compared to state-of-the-art methods.

Effectively models inter-modal dependencies and temporal correspondences.

Demonstrates the benefit of multimodal fusion in capturing video semantics.

Abstract

The rapid proliferation of online video content necessitates effective video summarization techniques. Traditional methods, often relying on a single modality (typically visual), struggle to capture the full semantic richness of videos. This paper introduces MF2Summ, a novel video summarization model based on multimodal content understanding, integrating both visual and auditory information. MF2Summ employs a five-stage process: feature extraction, cross-modal attention interaction, feature fusion, segment prediction, and key shot selection. Visual features are extracted using a pre-trained GoogLeNet model, while auditory features are derived using SoundNet. The core of our fusion mechanism involves a cross-modal Transformer and an alignment-guided self-attention Transformer, designed to effectively model inter-modal dependencies and temporal correspondences. Segment importance, location, and center-ness are predicted, followed by key shot selection using Non-Maximum Suppression (NMS) and the Kernel Temporal Segmentation (KTS) algorithm. Experimental results on the SumMe and TVSum datasets demonstrate that MF2Summ achieves competitive performance, notably improving F1-scores by 1.9\% and 0.6\% respectively over the DSNet model, and performing favorably against other state-of-the-art methods.