Prompts to Summaries: Zero-Shot Language-Guided Video Summarization with Large Language and Video Models

TL;DR

This paper presents Prompts-to-Summaries, a zero-shot, user-guided video summarization method using large language and video models, outperforming unsupervised methods without training data.

Contribution

It introduces a novel zero-shot framework that converts video captions into user-guided summaries using off-the-shelf models and large language models, without requiring training.

Findings

Surpasses all prior unsupervised methods on SumMe and TVSum datasets.

Performs competitively on the Query-Focused Video Summarization benchmark.

Provides a new dataset, VidSum-Reason, for long-tailed, multi-step reasoning video queries.

Abstract

The explosive growth of video data intensified the need for flexible user-controllable summarization tools that operate without training data. Existing methods either rely on domain-specific datasets, limiting generalization, or cannot incorporate user intent expressed in natural language. We introduce Prompts-to-Summaries: the first zero-shot, text-queryable video-summarizer that converts off-the-shelf video-language models (VidLMs) captions into user-guided skims via large-language-models (LLMs) judging, without the use of training data, beating unsupervised and matching supervised methods. Our pipeline (i) segments video into scenes, (ii) produces scene descriptions with a memory-efficient batch prompting scheme that scales to hours on a single GPU, (iii) scores scene importance with an LLM via tailored prompts, and (iv) propagates scores to frames using new consistency (temporal coherence) and uniqueness (novelty) metrics for fine-grained frame importance. On SumMe and TVSum, our approach surpasses all prior data-hungry unsupervised methods and performs competitively on the Query-Focused Video Summarization benchmark, where the competing methods require supervised frame-level importance. We release VidSum-Reason, a query-driven dataset featuring long-tailed concepts and multi-step reasoning, where our framework serves as the first challenging baseline. Overall, we demonstrate that pretrained multi-modal models, when orchestrated with principled prompting and score propagation, provide a powerful foundation for universal, text-queryable video summarization.