Towards Data-Efficient Video Pre-training with Frozen Image Foundation Models

TL;DR

This paper investigates a lightweight approach to video pre-training by freezing an image foundation model and training only a temporal module, aiming to reduce data and compute costs while maintaining strong performance.

Contribution

It introduces a novel paradigm of reusing pre-trained image models with a frozen spatial encoder and training a recurrent temporal module for video understanding.

Findings

Strong temporal performance achieved without large-scale video pre-training

Reusing image foundation models reduces data and compute requirements

Empirical results across multiple tasks support the approach's feasibility

Abstract

Video foundation models achieve strong performance across many video understanding tasks, but typically require large-scale pre-training on massive video datasets, resulting in substantial data and compute costs. In contrast, modern image foundation models already provide powerful spatial representations. This raises an important question: can competitive video models be built by reusing these spatial representations and pre-training only for temporal reasoning? We take initial steps toward exploring a lightweight training paradigm that freezes a pre-trained image foundation model and trains only a recurrent temporal module to process streaming video. By reusing an image foundation model as a spatial encoder, this approach could significantly reduce the amount of video data and compute required compared to end-to-end video pre-training. In this work, we explore the feasibility of this approach before investing in computing for video pre-training. Our empirical findings across multiple video understanding tasks suggest that strong temporal performance can emerge without large-scale video pre-training, motivating future work on recurrent video foundation models obtained by pre-training a temporal module on top of a frozen image foundation model. Code: https://github.com/tue-mps/towards-video-image-frozen .