DiverseVAR: Balancing Diversity and Quality of Next-Scale Visual Autoregressive Models

TL;DR

DiverseVAR is a test-time framework that improves the diversity of visual autoregressive models without retraining, using noise injection and a novel scale-travel refinement to balance diversity and image quality.

Contribution

It introduces a novel test-time method combining noise injection and scale-travel refinement to enhance diversity in VAR models without retraining.

Findings

Significantly increases diversity of generated images

Maintains high image quality despite increased diversity

Establishes a new Pareto frontier in diversity-quality trade-off

Abstract

We introduce DiverseVAR, a framework that enhances the diversity of text-conditioned visual autoregressive models (VAR) at test time without requiring retraining, fine-tuning, or substantial computational overhead. While VAR models have recently emerged as strong competitors to diffusion and flow models for image generation, they suffer from a critical limitation in diversity, often producing nearly identical images even for simple prompts. This issue has largely gone unnoticed amid the predominant focus on image quality. We address this limitation at test time in two stages. First, inspired by diversity enhancement techniques in diffusion models, we propose injecting noise into the text embedding. This introduces a trade-off between diversity and image quality: as diversity increases, the image quality sharply declines. To preserve quality, we propose scale-travel: a novel latent refinement technique inspired by time-travel strategies in diffusion models. Specifically, we use a multi-scale autoencoder to extract coarse-scale tokens that enable us to resume generation at intermediate stages. Extensive experiments show that combining text-embedding noise injection with our scale-travel refinement significantly enhances diversity while minimizing image-quality degradation, achieving a new Pareto frontier in the diversity-quality trade-off.