Query-Focused Extractive Video Summarization

TL;DR

This paper introduces SH-DPP, a probabilistic model for query-focused extractive video summarization that selects relevant key shots based on user queries and video importance, verified on annotated datasets.

Contribution

The paper proposes a novel Sequential and Hierarchical Determinantal Point Process model for query-focused video summarization, integrating relevance and importance in shot selection.

Findings

Effective in selecting relevant video shots based on user queries

Verified on densely annotated datasets with promising results

Enhances search engine capabilities for video snippet retrieval

Abstract

Video data is explosively growing. As a result of the "big video data", intelligent algorithms for automatic video summarization have re-emerged as a pressing need. We develop a probabilistic model, Sequential and Hierarchical Determinantal Point Process (SH-DPP), for query-focused extractive video summarization. Given a user query and a long video sequence, our algorithm returns a summary by selecting key shots from the video. The decision to include a shot in the summary depends on the shot's relevance to the user query and importance in the context of the video, jointly. We verify our approach on two densely annotated video datasets. The query-focused video summarization is particularly useful for search engines, e.g., to display snippets of videos.