VideoAR: Autoregressive Video Generation via Next-Frame & Scale Prediction

TL;DR

VideoAR introduces a scalable autoregressive framework for video generation that combines multi-scale prediction with novel techniques to improve long-term consistency, achieving state-of-the-art results with significantly reduced inference steps.

Contribution

It is the first large-scale autoregressive video generation model that effectively integrates multi-scale spatial-temporal modeling and stabilization techniques.

Findings

Achieves state-of-the-art FVD scores among autoregressive models.

Reduces inference steps by over 10 times compared to previous methods.

Performs competitively with diffusion models on VBench.

Abstract

Recent advances in video generation have been dominated by diffusion and flow-matching models, which produce high-quality results but remain computationally intensive and difficult to scale. In this work, we introduce VideoAR, the first large-scale Visual Autoregressive (VAR) framework for video generation that combines multi-scale next-frame prediction with autoregressive modeling. VideoAR disentangles spatial and temporal dependencies by integrating intra-frame VAR modeling with causal next-frame prediction, supported by a 3D multi-scale tokenizer that efficiently encodes spatio-temporal dynamics. To improve long-term consistency, we propose Multi-scale Temporal RoPE, Cross-Frame Error Correction, and Random Frame Mask, which collectively mitigate error propagation and stabilize temporal coherence. Our multi-stage pretraining pipeline progressively aligns spatial and temporal learning across increasing resolutions and durations. Empirically, VideoAR achieves new state-of-the-art results among autoregressive models, improving FVD on UCF-101 from 99.5 to 88.6 while reducing inference steps by over 10x, and reaching a VBench score of 81.74-competitive with diffusion-based models an order of magnitude larger. These results demonstrate that VideoAR narrows the performance gap between autoregressive and diffusion paradigms, offering a scalable, efficient, and temporally consistent foundation for future video generation research.