Diversity Has Always Been There in Your Visual Autoregressive Models

TL;DR

DiverseVAR enhances the diversity of Visual Autoregressive models by manipulating feature maps, significantly improving output variability without additional training or compromising image quality.

Contribution

The paper introduces DiverseVAR, a simple method to restore diversity in VAR models by feature map manipulation, without extra training.

Findings

Substantially improves generative diversity in VAR models.

Maintains high-fidelity image synthesis.

Requires no additional training.

Abstract

Visual Autoregressive (VAR) models have recently garnered significant attention for their innovative next-scale prediction paradigm, offering notable advantages in both inference efficiency and image quality compared to traditional multi-step autoregressive (AR) and diffusion models. However, despite their efficiency, VAR models often suffer from the diversity collapse i.e., a reduction in output variability, analogous to that observed in few-step distilled diffusion models. In this paper, we introduce DiverseVAR, a simple yet effective approach that restores the generative diversity of VAR models without requiring any additional training. Our analysis reveals the pivotal component of the feature map as a key factor governing diversity formation at early scales. By suppressing the pivotal component in the model input and amplifying it in the model output, DiverseVAR effectively unlocks the inherent generative potential of VAR models while preserving high-fidelity synthesis. Empirical results demonstrate that our approach substantially enhances generative diversity with only neglectable performance influences. Our code will be publicly released at https://github.com/wangtong627/DiverseVAR.