Guiding Visual Autoregressive Models through Spectrum Weakening

TL;DR

This paper introduces a spectrum-weakening framework for visual autoregressive models that improves unconditional generation quality without retraining or architectural changes, by controlling spectral information in internal representations.

Contribution

It proposes a novel spectral domain manipulation method for AR models that enhances generation quality and condition alignment without model re-training or structural modifications.

Findings

Enables high-quality unconditional generation

Maintains strong prompt alignment for conditional tasks

Works on both discrete and continuous AR models

Abstract

Classifier-free guidance (CFG) has become a widely adopted and practical approach for enhancing generation quality and improving condition alignment. Recent studies have explored guidance mechanisms for unconditional generation, yet these approaches remain fundamentally tied to assumptions specific to diffusion models. In this work, we propose a spectrum-weakening framework for visual autoregressive (AR) models. This method works without the need for re-training, specific conditions, or any architectural modifications. It achieves this by constructing a controllable weak model in the spectral domain. We theoretically show that invertible spectral transformations preserve information, while selectively retaining only a subset of spectrum introduces controlled information reduction. Based on this insight, we perform spectrum selection along the channel dimension of internal representations, which avoids the structural constraints imposed by diffusion models. We further introduce two spectrum renormalization strategies that ensures numerical stability during the weakening process. Extensive experiments were conducted on both discrete and continuous AR models, with text or class conditioning. The results demonstrate that our method enables high-quality unconditional generation while maintaining strong prompt alignment for conditional generation.