VFRTok: Variable Frame Rates Video Tokenizer with Duration-Proportional Information Assumption

TL;DR

VFRTok introduces a variable frame rate video tokenizer based on a new information assumption, improving efficiency and fidelity in video generation by using fewer tokens and asymmetric training.

Contribution

The paper proposes the Duration-Proportional Information Assumption and a Transformer-based tokenizer with Partial RoPE, enabling efficient variable frame rate encoding and improved content modeling.

Findings

Achieves state-of-the-art generation fidelity.

Uses only 1/8 tokens of existing tokenizers.

Provides competitive reconstruction quality.

Abstract

Modern video generation frameworks based on Latent Diffusion Models suffer from inefficiencies in tokenization due to the Frame-Proportional Information Assumption. Existing tokenizers provide fixed temporal compression rates, causing the computational cost of the diffusion model to scale linearly with the frame rate. The paper proposes the Duration-Proportional Information Assumption: the upper bound on the information capacity of a video is proportional to the duration rather than the number of frames. Based on this insight, the paper introduces VFRTok, a Transformer-based video tokenizer, that enables variable frame rate encoding and decoding through asymmetric frame rate training between the encoder and decoder. Furthermore, the paper proposes Partial Rotary Position Embeddings (RoPE) to decouple position and content modeling, which groups correlated patches into unified tokens. The Partial RoPE effectively improves content-awareness, enhancing the video generation capability. Benefiting from the compact and continuous spatio-temporal representation, VFRTok achieves competitive reconstruction quality and state-of-the-art generation fidelity while using only 1/8 tokens compared to existing tokenizers. The code and weights are released at: https://github.com/KwaiVGI/VFRTok.