Frozen CLIP Models are Efficient Video Learners

TL;DR

This paper introduces an efficient method for video recognition that leverages frozen CLIP image features and a lightweight Transformer decoder to learn high-quality video representations without fine-tuning the backbone.

Contribution

The paper proposes a novel framework called Efficient Video Learning (EVL) that uses frozen CLIP features and a lightweight decoder for effective video recognition.

Findings

Achieves high accuracy on multiple video datasets.

Reduces training resources compared to traditional fine-tuning.

Demonstrates effective temporal feature extraction with local modules.

Abstract

Video recognition has been dominated by the end-to-end learning paradigm -- first initializing a video recognition model with weights of a pretrained image model and then conducting end-to-end training on videos. This enables the video network to benefit from the pretrained image model. However, this requires substantial computation and memory resources for finetuning on videos and the alternative of directly using pretrained image features without finetuning the image backbone leads to subpar results. Fortunately, recent advances in Contrastive Vision-Language Pre-training (CLIP) pave the way for a new route for visual recognition tasks. Pretrained on large open-vocabulary image-text pair data, these models learn powerful visual representations with rich semantics. In this paper, we present Efficient Video Learning (EVL) -- an efficient framework for directly training high-quality video recognition models with frozen CLIP features. Specifically, we employ a lightweight Transformer decoder and learn a query token to dynamically collect frame-level spatial features from the CLIP image encoder. Furthermore, we adopt a local temporal module in each decoder layer to discover temporal clues from adjacent frames and their attention maps. We show that despite being efficient to train with a frozen backbone, our models learn high quality video representations on a variety of video recognition datasets. Code is available at https://github.com/OpenGVLab/efficient-video-recognition.