Cap2Sum: Learning to Summarize Videos by Generating Captions

TL;DR

Cap2Sum introduces a weakly-supervised learning approach for video summarization by generating captions, leveraging large-scale dense caption datasets and a CLIP prior to improve performance and generalization, enabling zero-shot and fine-tuned summarization.

Contribution

The paper proposes Cap2Sum, a novel model that uses dense caption generation for video summarization and incorporates CLIP priors to enhance object recognition, enabling better generalization and zero-shot capabilities.

Findings

Significant performance improvements over previous methods.

Effective zero-shot video summarization demonstrated.

Introduction of new datasets TVSum-Caption and SumMe-Caption.

Abstract

With the rapid growth of video data on the internet, video summarization is becoming a very important AI technology. However, due to the high labelling cost of video summarization, existing studies have to be conducted on small-scale datasets, leading to limited performance and generalization capacity. In this work, we introduce the use of dense video captions as a supervision signal to train video summarization models. Motivated by this, we propose Cap2Sum, a model that learns to summarize videos by generating captions, to exploit dense video caption annotations. This weakly-supervised approach allows us to train the models on large-scale dense video caption datasets to achieve better performance and generalization capacity. To further improve the generalization capacity, we introduce a CLIP (a strong vision-language model) Prior mechanism to enhance the learning of important objects that captions may ignore in the videos. In practice, Cap2Sum can perform zero-shot video summarization or be fine-tuned by the ground-truth summary or video caption of the target dataset. To examine the performance of Cap2Sum after weakly-supervised fine-tuning by the video captions, we propose two new datasets, TVSum-Caption and SumMe-Caption, which are derived from two common video summarization datasets and will be publicly released. We conduct extensive experiments and the results demonstrate that our method achieves significant improvements in performance and generalization capacity compared with previous methods.