Recognizing and Presenting the Storytelling Video Structure with Deep Multimodal Networks

TL;DR

This paper introduces a deep multimodal network approach for segmenting and understanding storytelling videos by combining perceptual, audio, and semantic cues to identify meaningful story boundaries and improve retrieval and summarization.

Contribution

It presents a novel deep network architecture that integrates multiple modalities for semantic video segmentation and proposes a retrieval strategy based on aesthetic and semantic value.

Findings

Effective segmentation of videos into meaningful stories.

Improved retrieval of significant video segments based on queries.

High agreement between automatic segmentation and human annotations.

Abstract

This paper presents a novel approach for temporal and semantic segmentation of edited videos into meaningful segments, from the point of view of the storytelling structure. The objective is to decompose a long video into more manageable sequences, which can in turn be used to retrieve the most significant parts of it given a textual query and to provide an effective summarization. Previous video decomposition methods mainly employed perceptual cues, tackling the problem either as a story change detection, or as a similarity grouping task, and the lack of semantics limited their ability to identify story boundaries. Our proposal connects together perceptual, audio and semantic cues in a specialized deep network architecture designed with a combination of CNNs which generate an appropriate embedding, and clusters shots into connected sequences of semantic scenes, i.e. stories. A retrieval presentation strategy is also proposed, by selecting the semantically and aesthetically "most valuable" thumbnails to present, considering the query in order to improve the storytelling presentation. Finally, the subjective nature of the task is considered, by conducting experiments with different annotators and by proposing an algorithm to maximize the agreement between automatic results and human annotators.