Contrastive Losses Are Natural Criteria for Unsupervised Video Summarization

TL;DR

This paper introduces a contrastive loss-based approach for unsupervised video summarization, directly quantifying frame importance without heuristic objectives, leading to high-quality summaries with minimal training.

Contribution

It proposes three contrastive metrics for importance scoring and demonstrates that pre-trained features, refined with contrastive learning, outperform previous methods.

Findings

High-quality importance scores from pre-trained features.

Refined features improve summary quality.

Method generalizes well to unseen videos.

Abstract

Video summarization aims to select the most informative subset of frames in a video to facilitate efficient video browsing. Unsupervised methods usually rely on heuristic training objectives such as diversity and representativeness. However, such methods need to bootstrap the online-generated summaries to compute the objectives for importance score regression. We consider such a pipeline inefficient and seek to directly quantify the frame-level importance with the help of contrastive losses in the representation learning literature. Leveraging the contrastive losses, we propose three metrics featuring a desirable key frame: local dissimilarity, global consistency, and uniqueness. With features pre-trained on the image classification task, the metrics can already yield high-quality importance scores, demonstrating competitive or better performance than past heavily-trained methods. We show that by refining the pre-trained features with a lightweight contrastively learned projection module, the frame-level importance scores can be further improved, and the model can also leverage a large number of random videos and generalize to test videos with decent performance. Code available at https://github.com/pangzss/pytorch-CTVSUM.