Unsupervised Video Highlight Detection by Learning from Audio and Visual Recurrence

TL;DR

This paper introduces an unsupervised method for video highlight detection that leverages recurrence patterns in both audio and visual data across videos, eliminating the need for manual annotations and outperforming prior approaches.

Contribution

It proposes a novel unsupervised approach that combines audio and visual recurrence to identify highlights without manual labels, utilizing pseudo-categories and clustering techniques.

Findings

Outperforms prior unsupervised methods on three benchmarks.

Effectively utilizes audio features alongside visual data.

Demonstrates the importance of audio in highlight detection.

Abstract

With the exponential growth of video content, the need for automated video highlight detection to extract key moments or highlights from lengthy videos has become increasingly pressing. This technology has the potential to enhance user experiences by allowing quick access to relevant content across diverse domains. Existing methods typically rely either on expensive manually labeled frame-level annotations, or on a large external dataset of videos for weak supervision through category information. To overcome this, we focus on unsupervised video highlight detection, eliminating the need for manual annotations. We propose a novel unsupervised approach which capitalizes on the premise that significant moments tend to recur across multiple videos of the similar category in both audio and visual modalities. Surprisingly, audio remains under-explored, especially in unsupervised algorithms, despite its potential to detect key moments. Through a clustering technique, we identify pseudo-categories of videos and compute audio pseudo-highlight scores for each video by measuring the similarities of audio features among audio clips of all the videos within each pseudo-category. Similarly, we also compute visual pseudo-highlight scores for each video using visual features. Then, we combine audio and visual pseudo-highlights to create the audio-visual pseudo ground-truth highlight of each video for training an audio-visual highlight detection network. Extensive experiments and ablation studies on three benchmarks showcase the superior performance of our method over prior work.