Fine-Grained Captioning of Long Videos through Scene Graph Consolidation

TL;DR

This paper presents a novel graph consolidation framework for long video captioning that generates detailed, coherent descriptions without additional fine-tuning, outperforming existing methods in zero-shot settings.

Contribution

Introduces a graph consolidation approach that combines segment-level captions into a unified representation for long videos, enhancing temporal understanding without extra fine-tuning.

Findings

Outperforms existing LLM-based consolidation methods.

Achieves strong zero-shot performance.

Reduces computational costs significantly.

Abstract

Recent advances in vision-language models have led to impressive progress in caption generation for images and short video clips. However, these models remain constrained by their limited temporal receptive fields, making it difficult to produce coherent and comprehensive captions for long videos. While several methods have been proposed to aggregate information across video segments, they often rely on supervised fine-tuning or incur significant computational overhead. To address these challenges, we introduce a novel framework for long video captioning based on graph consolidation. Our approach first generates segment-level captions, corresponding to individual frames or short video intervals, using off-the-shelf visual captioning models. These captions are then parsed into individual scene graphs, which are subsequently consolidated into a unified graph representation that preserves both holistic context and fine-grained details throughout the video. A lightweight graph-to-text decoder then produces the final video-level caption. This framework effectively extends the temporal understanding capabilities of existing models without requiring any additional fine-tuning on long video datasets. Experimental results show that our method significantly outperforms existing LLM-based consolidation approaches, achieving strong zero-shot performance while substantially reducing computational costs.