It's Just Another Day: Unique Video Captioning by Discriminative Prompting

TL;DR

This paper introduces a novel method called Captioning by Discriminative Prompting (CDP) to generate unique video captions for clips with identical descriptions, improving retrieval accuracy in videos with repetitive content.

Contribution

The paper proposes a new approach, CDP, that predicts discriminative properties to produce unique captions for similar video clips, along with two new benchmarks for evaluation.

Findings

CDP improves text-to-video R@1 by 15% on egocentric videos.

CDP enhances caption uniqueness in timeloop movies by 10%.

Introduces benchmarks for unique captioning in repetitive video scenarios.

Abstract

Long videos contain many repeating actions, events and shots. These repetitions are frequently given identical captions, which makes it difficult to retrieve the exact desired clip using a text search. In this paper, we formulate the problem of unique captioning: Given multiple clips with the same caption, we generate a new caption for each clip that uniquely identifies it. We propose Captioning by Discriminative Prompting (CDP), which predicts a property that can separate identically captioned clips, and use it to generate unique captions. We introduce two benchmarks for unique captioning, based on egocentric footage and timeloop movies - where repeating actions are common. We demonstrate that captions generated by CDP improve text-to-video R@1 by 15% for egocentric videos and 10% in timeloop movies.