Recurrent Video Masked Autoencoders

TL;DR

Recurrent Video Masked Autoencoders (RVM) introduce a recurrent transformer-based model for efficient, long-term video representation learning, achieving competitive performance with state-of-the-art models on various video tasks.

Contribution

RVM is a novel recurrent transformer-based video encoder that improves efficiency and long-term feature propagation without requiring knowledge distillation.

Findings

RVM matches or exceeds state-of-the-art on action classification and tracking.

RVM achieves up to 30x greater parameter efficiency than competing models.

RVM effectively propagates features over long temporal horizons with linear cost.

Abstract

We present Recurrent Video Masked-Autoencoders (RVM): a novel approach to video representation learning that leverages recurrent computation to model the temporal structure of video data. RVM couples an asymmetric masking objective with a transformer-based recurrent neural network to aggregate information over time, training solely on a simple pixel reconstruction loss. This design yields a highly efficient "generalist" encoder: RVM achieves competitive performance with state-of-the-art video models (e.g. VideoMAE, V-JEPA) on video-level tasks like action classification, and point and object tracking, while matching or exceeding the performance of image models (e.g. DINOv2) on tasks that require strong geometric and dense spatial features. Notably, RVM achieves strong performance in the small-model regime without requiring knowledge distillation, exhibiting up to 30x greater parameter efficiency than competing video masked autoencoders. Finally, we demonstrate that RVM's recurrent nature allows for stable feature propagation over long temporal horizons with linear computational cost, overcoming some of the limitations of standard spatio-temporal attention-based video models. Ablation studies further highlight the factors driving the model's success, with qualitative results showing that RVM learns rich representations of scene semantics, structure, and motion.