CLIP4Caption: CLIP for Video Caption

TL;DR

This paper introduces CLIP4Caption, a framework that enhances video captioning by leveraging CLIP for better visual-text alignment and employs a Transformer decoder with an ensemble strategy, achieving state-of-the-art results.

Contribution

The paper proposes a CLIP-enhanced video-text matching network and a Transformer-based decoder, along with a novel ensemble strategy, to improve video captioning performance.

Findings

Achieved up to 10% improvement in CIDEr score on MSR-VTT dataset.

Secured 2nd place in ACM MM 2021 Video Understanding Challenge.

Demonstrated effectiveness with state-of-the-art results on two datasets.

Abstract

Video captioning is a challenging task since it requires generating sentences describing various diverse and complex videos. Existing video captioning models lack adequate visual representation due to the neglect of the existence of gaps between videos and texts. To bridge this gap, in this paper, we propose a CLIP4Caption framework that improves video captioning based on a CLIP-enhanced video-text matching network (VTM). This framework is taking full advantage of the information from both vision and language and enforcing the model to learn strongly text-correlated video features for text generation. Besides, unlike most existing models using LSTM or GRU as the sentence decoder, we adopt a Transformer structured decoder network to effectively learn the long-range visual and language dependency. Additionally, we introduce a novel ensemble strategy for captioning tasks. Experimental results demonstrate the effectiveness of our method on two datasets: 1) on MSR-VTT dataset, our method achieved a new state-of-the-art result with a significant gain of up to 10% in CIDEr; 2) on the private test data, our method ranking 2nd place in the ACM MM multimedia grand challenge 2021: Pre-training for Video Understanding Challenge. It is noted that our model is only trained on the MSR-VTT dataset.