Extending Video Masked Autoencoders to 128 frames

TL;DR

This paper introduces a novel adaptive masking strategy for Video Masked Autoencoders, enabling effective pre-training on longer videos (128 frames) and achieving superior performance over shorter video models.

Contribution

It proposes an adaptive decoder masking method that prioritizes important tokens and uses a MAGVIT-based tokenizer, allowing training on longer videos with improved results.

Findings

Outperforms short-video models on Diving48 and EPIC-Kitchens-100

Enables training on 128-frame videos with better accuracy

Uses a simple architecture with video-only pre-training

Abstract

Video understanding has witnessed significant progress with recent video foundation models demonstrating strong performance owing to self-supervised pre-training objectives; Masked Autoencoders (MAE) being the design of choice. Nevertheless, the majority of prior works that leverage MAE pre-training have focused on relatively short video representations (16 / 32 frames in length) largely due to hardware memory and compute limitations that scale poorly with video length due to the dense memory-intensive self-attention decoding. One natural strategy to address these challenges is to subsample tokens to reconstruct during decoding (or decoder masking). In this work, we propose an effective strategy for prioritizing tokens which allows training on longer video sequences (128 frames) and gets better performance than, more typical, random and uniform masking strategies. The core of our approach is an adaptive decoder masking strategy that prioritizes the most important tokens and uses quantized tokens as reconstruction objectives. Our adaptive strategy leverages a powerful MAGVIT-based tokenizer that jointly learns the tokens and their priority. We validate our design choices through exhaustive ablations and observe improved performance of the resulting long-video (128 frames) encoders over short-video (32 frames) counterparts. With our long-video masked autoencoder (LVMAE) strategy, we surpass state-of-the-art on Diving48 by 3.9 points and EPIC-Kitchens-100 verb classification by 2.5 points while relying on a simple core architecture and video-only pre-training (unlike some of the prior works that require millions of labeled video-text pairs or specialized encoders).