Markovian Scale Prediction: A New Era of Visual Autoregressive Generation

TL;DR

This paper introduces Markov-VAR, a new visual autoregressive model that improves efficiency and performance by reformulating the process as a Markovian system with a sliding window, reducing computational costs while maintaining effectiveness.

Contribution

The paper proposes Markov-VAR, a novel non-full-context Markov process formulation for visual autoregressive generation that enhances efficiency and scalability over traditional VAR models.

Findings

Reduces FID by 10.5% on ImageNet

Decreases peak memory consumption by 83.8%

Achieves comparable or better generation quality with less computational cost

Abstract

Visual AutoRegressive modeling (VAR) based on next-scale prediction has revitalized autoregressive visual generation. Although its full-context dependency, i.e., modeling all previous scales for next-scale prediction, facilitates more stable and comprehensive representation learning by leveraging complete information flow, the resulting computational inefficiency and substantial overhead severely hinder VAR's practicality and scalability. This motivates us to develop a new VAR model with better performance and efficiency without full-context dependency. To address this, we reformulate VAR as a non-full-context Markov process, proposing Markov-VAR. It is achieved via Markovian Scale Prediction: we treat each scale as a Markov state and introduce a sliding window that compresses certain previous scales into a compact history vector to compensate for historical information loss owing to non-full-context dependency. Integrating the history vector with the Markov state yields a representative dynamic state that evolves under a Markov process. Extensive experiments demonstrate that Markov-VAR is extremely simple yet highly effective: Compared to VAR on ImageNet, Markov-VAR reduces FID by 10.5% (256 $\times$ 256) and decreases peak memory consumption by 83.8% (1024 $\times$ 1024). We believe that Markov-VAR can serve as a foundation for future research on visual autoregressive generation and other downstream tasks.