RefTok: Reference-Based Tokenization for Video Generation

TL;DR

RefTok introduces a reference-based tokenization approach that effectively captures temporal dependencies in videos, significantly improving compression and generation quality across multiple datasets compared to existing methods.

Contribution

RefTok is a novel reference-based tokenization method that encodes and decodes video frames conditioned on an unquantized reference, enhancing temporal modeling and compression.

Findings

Outperforms state-of-the-art tokenizers by 36.7% in metrics.

Improves video generation quality by 27.9% over larger models.

Maintains details and continuity across diverse video datasets.

Abstract

Effectively handling temporal redundancy remains a key challenge in learning video models. Prevailing approaches often treat each set of frames independently, failing to effectively capture the temporal dependencies and redundancies inherent in videos. To address this limitation, we introduce RefTok, a novel reference-based tokenization method capable of capturing complex temporal dynamics and contextual information. Our method encodes and decodes sets of frames conditioned on an unquantized reference frame. When decoded, RefTok preserves the continuity of motion and the appearance of objects across frames. For example, RefTok retains facial details despite head motion, reconstructs text correctly, preserves small patterns, and maintains the legibility of handwriting from the context. Across 4 video datasets (K600, UCF-101, BAIR Robot Pushing, and DAVIS), RefTok significantly outperforms current state-of-the-art tokenizers (Cosmos and MAGVIT) and improves all evaluated metrics (PSNR, SSIM, LPIPS) by an average of 36.7% at the same or higher compression ratios. When a video generation model is trained using RefTok's latents on the BAIR Robot Pushing task, the generations not only outperform MAGVIT-B but the larger MAGVIT-L, which has 4x more parameters, across all generation metrics by an average of 27.9%.