Video Summarization via Actionness Ranking

TL;DR

This paper introduces a novel video summarization method that leverages the concept of actionness, or the intensity of deliberate motion, to improve the quality of generated summaries, validated through user studies and benchmark tests.

Contribution

It proposes a new approach that uses actionness ranking to guide video summary generation, differing from prior heuristic or complexity-based methods.

Findings

Actionness correlates strongly with human summary inclusion decisions.

The proposed method outperforms state-of-the-art summarization techniques.

User studies confirm the reliability of actionness as a key cue.

Abstract

To automatically produce a brief yet expressive summary of a long video, an automatic algorithm should start by resembling the human process of summary generation. Prior work proposed supervised and unsupervised algorithms to train models for learning the underlying behavior of humans by increasing modeling complexity or craft-designing better heuristics to simulate human summary generation process. In this work, we take a different approach by analyzing a major cue that humans exploit for the summary generation; the nature and intensity of actions. We empirically observed that a frame is more likely to be included in human-generated summaries if it contains a substantial amount of deliberate motion performed by an agent, which is referred to as actionness. Therefore, we hypothesize that learning to automatically generate summaries involves an implicit knowledge of actionness estimation and ranking. We validate our hypothesis by running a user study that explores the correlation between human-generated summaries and actionness ranks. We also run a consensus and behavioral analysis between human subjects to ensure reliable and consistent results. The analysis exhibits a considerable degree of agreement among subjects within obtained data and verifying our initial hypothesis. Based on the study findings, we develop a method to incorporate actionness data to explicitly regulate a learning algorithm that is trained for summary generation. We assess the performance of our approach to four summarization benchmark datasets and demonstrate an evident advantage compared to state-of-the-art summarization methods.