Scaling Up Video Summarization Pretraining with Large Language Models

TL;DR

This paper introduces a large-scale dataset and a new model for video summarization, leveraging large language models to improve generalization and set new state-of-the-art results.

Contribution

It presents an automated pipeline for creating a large video summarization dataset using LLMs and proposes a novel model that outperforms existing methods.

Findings

Achieved state-of-the-art results on multiple benchmarks.

Created a new benchmark dataset with professional annotations.

Demonstrated the effectiveness of LLMs in generating training data.

Abstract

Long-form video content constitutes a significant portion of internet traffic, making automated video summarization an essential research problem. However, existing video summarization datasets are notably limited in their size, constraining the effectiveness of state-of-the-art methods for generalization. Our work aims to overcome this limitation by capitalizing on the abundance of long-form videos with dense speech-to-video alignment and the remarkable capabilities of recent large language models (LLMs) in summarizing long text. We introduce an automated and scalable pipeline for generating a large-scale video summarization dataset using LLMs as Oracle summarizers. By leveraging the generated dataset, we analyze the limitations of existing approaches and propose a new video summarization model that effectively addresses them. To facilitate further research in the field, our work also presents a new benchmark dataset that contains 1200 long videos each with high-quality summaries annotated by professionals. Extensive experiments clearly indicate that our proposed approach sets a new state-of-the-art in video summarization across several benchmarks.