FRAME: Pre-Training Video Feature Representations via Anticipation and Memory

TL;DR

FRAME is a self-supervised video encoder that predicts future features from past frames, achieving spatially precise, temporally coherent dense representations and outperforming existing models on multiple video understanding tasks.

Contribution

It introduces a novel self-supervised training method that leverages image-based models for dense video prediction, bridging the gap between image and video encoders.

Findings

Outperforms existing self-supervised video models on six dense prediction tasks.

Leverages image-based models for dense video understanding with superior results.

Maintains a compact architecture suitable for various downstream applications.

Abstract

Dense video prediction tasks, such as object tracking and semantic segmentation, require video encoders that generate temporally consistent, spatially dense features for every frame. However, existing approaches fall short: image encoders like DINO or CLIP lack temporal awareness, while video models such as VideoMAE underperform compared to image encoders on dense prediction tasks. We address this gap with FRAME, a self-supervised video frame encoder tailored for dense video understanding. FRAME learns to predict current and future DINO patch features from past and present RGB frames, leading to spatially precise and temporally coherent representations. To our knowledge, FRAME is the first video encoder to leverage image-based models for dense prediction while outperforming them on tasks requiring fine-grained visual correspondence. As an auxiliary capability, FRAME aligns its class token with CLIP's semantic space, supporting language-driven tasks such as video classification. We evaluate FRAME across six dense prediction tasks on seven datasets, where it consistently outperforms image encoders and existing self-supervised video models. Despite its versatility, FRAME maintains a compact architecture suitable for a range of downstream applications.