SGCap: Decoding Semantic Group for Zero-shot Video Captioning

TL;DR

SGCap introduces a novel zero-shot video captioning approach that models temporal dynamics and semantic diversity, significantly improving performance over previous methods and rivaling fully supervised models.

Contribution

It proposes the Semantic Group Decoding strategy along with Key Sentences Selection and Probability Sampling Supervision modules for effective zero-shot video captioning.

Findings

Outperforms previous zero-shot methods on benchmarks.

Achieves performance competitive with fully supervised models.

Effectively models temporal dynamics and semantic diversity.

Abstract

Zero-shot video captioning aims to generate sentences for describing videos without training the model on video-text pairs, which remains underexplored. Existing zero-shot image captioning methods typically adopt a text-only training paradigm, where a language decoder reconstructs single-sentence embeddings obtained from CLIP. However, directly extending them to the video domain is suboptimal, as applying average pooling over all frames neglects temporal dynamics. To address this challenge, we propose a Semantic Group Captioning (SGCap) method for zero-shot video captioning. In particular, it develops the Semantic Group Decoding (SGD) strategy to employ multi-frame information while explicitly modeling inter-frame temporal relationships. Furthermore, existing zero-shot captioning methods that rely on cosine similarity for sentence retrieval and reconstruct the description supervised by a single frame-level caption, fail to provide sufficient video-level supervision. To alleviate this, we introduce two key components, including the Key Sentences Selection (KSS) module and the Probability Sampling Supervision (PSS) module. The two modules construct semantically-diverse sentence groups that models temporal dynamics and guide the model to capture inter-sentence causal relationships, thereby enhancing its generalization ability to video captioning. Experimental results on several benchmarks demonstrate that SGCap significantly outperforms previous state-of-the-art zero-shot alternatives and even achieves performance competitive with fully supervised ones. Code is available at https://github.com/mlvccn/SGCap_Video.