Building an Open-Vocabulary Video CLIP Model with Better Architectures, Optimization and Data

TL;DR

This paper introduces Open-VCLIP++, a framework that adapts CLIP for zero-shot video recognition by capturing spatial-temporal relationships and leveraging large language models, achieving state-of-the-art results across multiple datasets.

Contribution

The paper proposes Open-VCLIP++, a novel adaptation of CLIP for zero-shot video classification, incorporating weight interpolation and fine-grained video descriptions for improved generalization.

Findings

Achieves zero-shot accuracy of 88.1% on UCF dataset.

Outperforms existing methods by over 8% on multiple datasets.

Demonstrates effective transfer of CLIP to video domain with less fine-tuning data.

Abstract

Despite significant results achieved by Contrastive Language-Image Pretraining (CLIP) in zero-shot image recognition, limited effort has been made exploring its potential for zero-shot video recognition. This paper presents Open-VCLIP++, a simple yet effective framework that adapts CLIP to a strong zero-shot video classifier, capable of identifying novel actions and events during testing. Open-VCLIP++ minimally modifies CLIP to capture spatial-temporal relationships in videos, thereby creating a specialized video classifier while striving for generalization. We formally demonstrate that training Open-VCLIP++ is tantamount to continual learning with zero historical data. To address this problem, we introduce Interpolated Weight Optimization, a technique that leverages the advantages of weight interpolation during both training and testing. Furthermore, we build upon large language models to produce fine-grained video descriptions. These detailed descriptions are further aligned with video features, facilitating a better transfer of CLIP to the video domain. Our approach is evaluated on three widely used action recognition datasets, following a variety of zero-shot evaluation protocols. The results demonstrate that our method surpasses existing state-of-the-art techniques by significant margins. Specifically, we achieve zero-shot accuracy scores of 88.1%, 58.7%, and 81.2% on UCF, HMDB, and Kinetics-600 datasets respectively, outpacing the best-performing alternative methods by 8.5%, 8.2%, and 12.3%. We also evaluate our approach on the MSR-VTT video-text retrieval dataset, where it delivers competitive video-to-text and text-to-video retrieval performance, while utilizing substantially less fine-tuning data compared to other methods. Code is released at https://github.com/wengzejia1/Open-VCLIP.