PyraTok: Language-Aligned Pyramidal Tokenizer for Video Understanding and Generation

TL;DR

PyraTok is a novel pyramidal tokenizer that learns multi-scale, semantically structured video tokens aligned with language, significantly enhancing zero-shot video understanding and generation across various benchmarks.

Contribution

It introduces LaPQ, a new multi-scale quantization method, and demonstrates improved cross-modal alignment and state-of-the-art zero-shot performance in video tasks.

Findings

Achieves state-of-the-art video reconstruction results.

Improves text-to-video quality across benchmarks.

Sets new zero-shot performance records in video segmentation and action localization.

Abstract

Discrete video VAEs underpin modern text-to-video generation and video understanding systems, yet existing tokenizers typically learn visual codebooks at a single scale with limited vocabularies and shallow language supervision, leading to poor cross-modal alignment and zero-shot transfer. We introduce PyraTok, a language-aligned pyramidal tokenizer that learns semantically structured discrete latents across multiple spatiotemporal resolutions. PyraTok builds on a pretrained video VAE and a novel Language aligned Pyramidal Quantization (LaPQ) module that discretizes encoder features at several depths using a shared large binary codebook, yielding compact yet expressive video token sequences. To tightly couple visual tokens with language, PyraTok jointly optimizes multi-scale text-guided quantization and a global autoregressive objective over the token hierarchy. Across ten benchmarks, PyraTok delivers state-of-the-art (SOTA) video reconstruction, consistently improves text-to-video quality, and sets new SOTA zero-shot performance on video segmentation, temporal action localization, and video understanding, scaling robustly to up to 4K/8K resolutions.