On the Evaluation of Video Keyframe Summaries using User Ground Truth

TL;DR

This paper introduces a formal evaluation method for video keyframe summaries using user ground truth, proposing a discrimination capacity measure to quantify improvements over baselines and validating it with various features and matching methods.

Contribution

It develops a formal protocol and a discrimination capacity measure for evaluating video summaries, addressing limitations of subjective user studies.

Findings

Hue histograms are effective for summary comparison.

Discrimination capacity correlates with user preferences.

Proposed protocol enables consistent evaluation with ground truth.

Abstract

Given the great interest in creating keyframe summaries from video, it is surprising how little has been done to formalise their evaluation and comparison. User studies are often carried out to demonstrate that a proposed method generates a more appealing summary than one or two rival methods. But larger comparison studies cannot feasibly use such user surveys. Here we propose a discrimination capacity measure as a formal way to quantify the improvement over the uniform baseline, assuming that one or more ground truth summaries are available. Using the VSUMM video collection, we examine 10 video feature types, including CNN and SURF, and 6 methods for matching frames from two summaries. Our results indicate that a simple frame representation through hue histograms suffices for the purposes of comparing keyframe summaries. We subsequently propose a formal protocol for comparing summaries when ground truth is available.