Follow the Saliency: Supervised Saliency for Retrieval-augmented Dense Video Captioning

TL;DR

The paper introduces STaRC, a supervised saliency framework for dense video captioning that improves temporal segmentation accuracy and caption relevance by leveraging ground truth event boundaries without extra annotations.

Contribution

STaRC employs a highlight detection module trained on existing annotations to guide saliency-driven segmentation and captioning, achieving state-of-the-art results.

Findings

Achieves superior performance on YouCook2 and ViTT benchmarks.

Produces temporally coherent segments aligned with event boundaries.

Enhances caption relevance through saliency-guided prompts.

Abstract

Existing retrieval-augmented approaches for Dense Video Captioning (DVC) often fail to achieve accurate temporal segmentation aligned with true event boundaries, as they rely on heuristic strategies that overlook ground truth event boundaries. The proposed framework, \textbf{STaRC}, overcomes this limitation by supervising frame-level saliency through a highlight detection module. Note that the highlight detection module is trained on binary labels derived directly from DVC ground truth annotations without the need for additional annotation. We also propose to utilize the saliency scores as a unified temporal signal that drives retrieval via saliency-guided segmentation and informs caption generation through explicit Saliency Prompts injected into the decoder. By enforcing saliency-constrained segmentation, our method produces temporally coherent segments that align closely with actual event transitions, leading to more accurate retrieval and contextually grounded caption generation. We conduct comprehensive evaluations on the YouCook2 and ViTT benchmarks, where STaRC achieves state-of-the-art performance across most of the metrics. Our code is available at https://github.com/ermitaju1/STaRC