ARVideo: Autoregressive Pretraining for Self-Supervised Video Representation Learning

TL;DR

ARVideo introduces a novel autoregressive self-supervised learning framework for videos, leveraging spatiotemporal token clustering and randomized prediction order to improve efficiency and performance.

Contribution

It proposes a new autoregressive video pretraining method with spatiotemporal clustering and randomized prediction order, enhancing contextual learning and training efficiency.

Findings

Achieves 81.2% on Kinetics-400 with ViT-B backbone.

Attains 70.9% on Something-Something V2.

Trains 14% faster and uses 58% less GPU memory than VideoMAE.

Abstract

This paper presents a new self-supervised video representation learning framework, ARVideo, which autoregressively predicts the next video token in a tailored sequence order. Two key designs are included. First, we organize autoregressive video tokens into clusters that span both spatially and temporally, thereby enabling a richer aggregation of contextual information compared to the standard spatial-only or temporal-only clusters. Second, we adopt a randomized spatiotemporal prediction order to facilitate learning from multi-dimensional data, addressing the limitations of a handcrafted spatial-first or temporal-first sequence order. Extensive experiments establish ARVideo as an effective paradigm for self-supervised video representation learning. For example, when trained with the ViT-B backbone, ARVideo competitively attains 81.2% on Kinetics-400 and 70.9% on Something-Something V2, which are on par with the strong benchmark set by VideoMAE. Importantly, ARVideo also demonstrates higher training efficiency, i.e., it trains 14% faster and requires 58% less GPU memory compared to VideoMAE.