# DistInit: Learning Video Representations Without a Single Labeled Video

## TL;DR

DistInit introduces a novel method for learning video representations without labeled videos by distilling knowledge from image-based models, enabling effective spatiotemporal feature learning from raw, unlabeled video data.

## Contribution

The paper presents a new approach that leverages image-based models as teachers to train video models without labeled videos, demonstrating effective spatiotemporal feature learning.

## Key findings

- Outperforms standard image-to-video bootstrapping techniques by 16%.
- Learns robust spatiotemporal features from uncurated raw video data.
- Works across different input modalities and teacher models.

## Abstract

Video recognition models have progressed significantly over the past few years, evolving from shallow classifiers trained on hand-crafted features to deep spatiotemporal networks. However, labeled video data required to train such models have not been able to keep up with the ever-increasing depth and sophistication of these networks. In this work, we propose an alternative approach to learning video representations that require no semantically labeled videos and instead leverages the years of effort in collecting and labeling large and clean still-image datasets. We do so by using state-of-the-art models pre-trained on image datasets as "teachers" to train video models in a distillation framework. We demonstrate that our method learns truly spatiotemporal features, despite being trained only using supervision from still-image networks. Moreover, it learns good representations across different input modalities, using completely uncurated raw video data sources and with different 2D teacher models. Our method obtains strong transfer performance, outperforming standard techniques for bootstrapping video architectures with image-based models by 16%. We believe that our approach opens up new approaches for learning spatiotemporal representations from unlabeled video data.