Faster than real-time detection of shot boundaries, sampling structure and dynamic keyframes in video

TL;DR

This paper introduces a fast, unified algorithm for detecting shot boundaries, sampling structure, and dynamic keyframes in videos, achieving real-time performance with high robustness to challenging content.

Contribution

The paper presents a novel, unified algorithm that performs multiple video analysis tasks efficiently and robustly, running four times faster than real-time.

Findings

Runs four times faster than real-time

Robust against challenging video content

Effective in detecting shot boundaries and keyframes

Abstract

The detection of shot boundaries (hardcuts and short dissolves), sampling structure (progressive / interlaced / pulldown) and dynamic keyframes in a video are fundamental video analysis tasks which have to be done before any further high-level analysis tasks. We present a novel algorithm which does all these analysis tasks in an unified way, by utilizing a combination of inter-frame and intra-frame measures derived from the motion field and normalized cross correlation. The algorithm runs four times faster than real-time due to sparse and selective calculation of these measures. An initial evaluation furthermore shows that the proposed algorithm is extremely robust even for challenging content showing large camera or object motion, flashlights, flicker or low contrast / noise.