LSRS: Latent Scale Rejection Sampling for Visual Autoregressive Modeling

TL;DR

This paper introduces LSRS, a method that refines latent token maps during inference in visual autoregressive models, significantly improving image generation quality with minimal extra computation.

Contribution

We propose Latent Scale Rejection Sampling (LSRS), a novel inference technique that enhances VAR models by refining token maps to reduce structural errors.

Findings

LSRS reduces FID score from 1.95 to 1.78 with only 1% increase in inference time.

LSRS further reduces FID to 1.66 with a 15% increase in inference time.

LSRS effectively mitigates autoregressive errors while maintaining computational efficiency.

Abstract

Visual Autoregressive (VAR) modeling approach for image generation proposes autoregressive processing across hierarchical scales, decoding multiple tokens per scale in parallel. This method achieves high-quality generation while accelerating synthesis. However, parallel token sampling within a scale may lead to structural errors, resulting in suboptimal generated images. To mitigate this, we propose Latent Scale Rejection Sampling (LSRS), a method that progressively refines token maps in the latent scale during inference to enhance VAR models. Our method uses a lightweight scoring model to evaluate multiple candidate token maps sampled at each scale, selecting the high-quality map to guide subsequent scale generation. By prioritizing early scales critical for structural coherence, LSRS effectively mitigates autoregressive error accumulation while maintaining computational efficiency. Experiments demonstrate that LSRS significantly improves VAR's generation quality with minimal additional computational overhead. For the VAR-d30 model, LSRS increases the inference time by merely 1% while reducing its FID score from 1.95 to 1.78. When the inference time is increased by 15%, the FID score can be further reduced to 1.66. LSRS offers an efficient test-time scaling solution for enhancing VAR-based generation.