Distilling Vision-Language Models on Millions of Videos

TL;DR

This paper introduces a method to create high-quality video captions by fine-tuning vision-language models with synthesized data, enabling improved video understanding and retrieval without extensive human-annotated datasets.

Contribution

We propose a novel video-instruction-tuning approach that leverages image-language models and auto-labeling to enhance video-language modeling and benchmark performance.

Findings

Outperforms previous methods on NExT-QA by 2.8%.

Achieves 6% improvement in zero-shot text-to-video retrieval on MSR-VTT.

Generates the largest video caption dataset to date.

Abstract

The recent advance in vision-language models is largely attributed to the abundance of image-text data. We aim to replicate this success for video-language models, but there simply is not enough human-curated video-text data available. We thus resort to fine-tuning a video-language model from a strong image-language baseline with synthesized instructional data. The resulting video model by video-instruction-tuning (VIIT) is then used to auto-label millions of videos to generate high-quality captions. We show the adapted video-language model performs well on a wide range of video-language benchmarks. For instance, it surpasses the best prior result on open-ended NExT-QA by 2.8%. Besides, our model generates detailed descriptions for previously unseen videos, which provide better textual supervision than existing methods. Experiments show that a video-language dual-encoder model contrastively trained on these auto-generated captions is 3.8% better than the strongest baseline that also leverages vision-language models. Our best model outperforms state-of-the-art methods on MSR-VTT zero-shot text-to-video retrieval by 6%. As a side product, we generate the largest video caption dataset to date.