OmniMAE: Single Model Masked Pretraining on Images and Videos

TL;DR

OmniMAE introduces a unified masked autoencoding approach for training a single Vision Transformer model on both images and videos, achieving state-of-the-art results without labeled data and enabling fast training with high sparsity.

Contribution

The paper demonstrates that a simple masked autoencoding method can effectively train a unified model for images and videos, outperforming previous multi-modal approaches.

Findings

Achieves 86.6% on ImageNet with a single model.

Sets new state-of-the-art at 75.5% on Something Something-v2.

Enables fast training by dropping up to 95% of patches.

Abstract

Transformer-based architectures have become competitive across a variety of visual domains, most notably images and videos. While prior work studies these modalities in isolation, having a common architecture suggests that one can train a single unified model for multiple visual modalities. Prior attempts at unified modeling typically use architectures tailored for vision tasks, or obtain worse performance compared to single modality models. In this work, we show that masked autoencoding can be used to train a simple Vision Transformer on images and videos, without requiring any labeled data. This single model learns visual representations that are comparable to or better than single-modality representations on both image and video benchmarks, while using a much simpler architecture. Furthermore, this model can be learned by dropping 90% of the image and 95% of the video patches, enabling extremely fast training of huge model architectures. In particular, we show that our single ViT-Huge model can be finetuned to achieve 86.6% on ImageNet and 75.5% on the challenging Something Something-v2 video benchmark, setting a new state-of-the-art.