Efficient Long Video Tokenization via Coordinate-based Patch Reconstruction

TL;DR

CoordTok introduces a coordinate-based video tokenizer that efficiently encodes long videos into fewer tokens, enabling memory-efficient training of long-video generative models.

Contribution

The paper presents CoordTok, a novel coordinate-based tokenizer that reduces token count for long videos, facilitating efficient training without high computational costs.

Findings

Reduces tokens for 128-frame videos from over 6000 to 1280.

Enables memory-efficient training of a 128-frame diffusion transformer.

Maintains high reconstruction quality with fewer tokens.

Abstract

Efficient tokenization of videos remains a challenge in training vision models that can process long videos. One promising direction is to develop a tokenizer that can encode long video clips, as it would enable the tokenizer to leverage the temporal coherence of videos better for tokenization. However, training existing tokenizers on long videos often incurs a huge training cost as they are trained to reconstruct all the frames at once. In this paper, we introduce CoordTok, a video tokenizer that learns a mapping from coordinate-based representations to the corresponding patches of input videos, inspired by recent advances in 3D generative models. In particular, CoordTok encodes a video into factorized triplane representations and reconstructs patches that correspond to randomly sampled $(x,y,t)$ coordinates. This allows for training large tokenizer models directly on long videos without requiring excessive training resources. Our experiments show that CoordTok can drastically reduce the number of tokens for encoding long video clips. For instance, CoordTok can encode a 128-frame video with 128$\times$128 resolution into 1280 tokens, while baselines need 6144 or 8192 tokens to achieve similar reconstruction quality. We further show that this efficient video tokenization enables memory-efficient training of a diffusion transformer that can generate 128 frames at once.