It Takes Two: Masked Appearance-Motion Modeling for Self-supervised Video Transformer Pre-training

TL;DR

This paper introduces MAM2, a self-supervised video pre-training framework that explicitly incorporates motion cues through a novel mask-and-predict approach, significantly improving training efficiency and representation quality.

Contribution

The work proposes a new Masked Appearance-Motion Modeling framework with a specialized encoder-regressor-decoder architecture for enhanced video representation learning.

Findings

Speeds up convergence, requiring half the epochs of previous methods.

Achieves state-of-the-art performance on multiple video benchmarks.

Effectively utilizes RGB-difference for motion prediction.

Abstract

Self-supervised video transformer pre-training has recently benefited from the mask-and-predict pipeline. They have demonstrated outstanding effectiveness on downstream video tasks and superior data efficiency on small datasets. However, temporal relation is not fully exploited by these methods. In this work, we explicitly investigate motion cues in videos as extra prediction target and propose our Masked Appearance-Motion Modeling (MAM2) framework. Specifically, we design an encoder-regressor-decoder pipeline for this task. The regressor separates feature encoding and pretext tasks completion, such that the feature extraction process is completed adequately by the encoder. In order to guide the encoder to fully excavate spatial-temporal features, two separate decoders are used for two pretext tasks of disentangled appearance and motion prediction. We explore various motion prediction targets and figure out RGB-difference is simple yet effective. As for appearance prediction, VQGAN codes are leveraged as prediction target. With our pre-training pipeline, convergence can be remarkably speed up, e.g., we only require half of epochs than state-of-the-art VideoMAE (400 v.s. 800) to achieve the competitive performance. Extensive experimental results prove that our method learns generalized video representations. Notably, our MAM2 with ViT-B achieves 82.3% on Kinects-400, 71.3% on Something-Something V2, 91.5% on UCF101, and 62.5% on HMDB51.