Towards an Automated Multimodal Approach for Video Summarization: Building a Bridge Between Text, Audio and Facial Cue-Based Summarization

TL;DR

This paper introduces a behaviour-aware multimodal video summarization framework that integrates text, audio, and visual cues to produce more accurate and emotionally relevant summaries, outperforming traditional methods.

Contribution

It presents a novel multimodal approach that combines prosodic, textual, and visual features, including the concept of bonus words, to enhance video summarization quality.

Findings

Significant improvements in ROUGE-1 and BERTScore metrics.

Enhanced F1-Score in video-based evaluation by nearly 23%.

Effective integration of multimodal cues for semantic and emotional relevance.

Abstract

The increasing volume of video content in educational, professional, and social domains necessitates effective summarization techniques that go beyond traditional unimodal approaches. This paper proposes a behaviour-aware multimodal video summarization framework that integrates textual, audio, and visual cues to generate timestamp-aligned summaries. By extracting prosodic features, textual cues and visual indicators, the framework identifies semantically and emotionally important moments. A key contribution is the identification of bonus words, which are terms emphasized across multiple modalities and used to improve the semantic relevance and expressive clarity of the summaries. The approach is evaluated against pseudo-ground truth (pGT) summaries generated using LLM-based extractive method. Experimental results demonstrate significant improvements over traditional extractive method, such as the Edmundson method, in both text and video-based evaluation metrics. Text-based metrics show ROUGE-1 increasing from 0.4769 to 0.7929 and BERTScore from 0.9152 to 0.9536, while in video-based evaluation, our proposed framework improves F1-Score by almost 23%. The findings underscore the potential of multimodal integration in producing comprehensive and behaviourally informed video summaries.