Rethinking Training Dynamics in Scale-wise Autoregressive Generation

TL;DR

This paper identifies key training challenges in scale-wise autoregressive models and proposes Self-Autoregressive Refinement (SAR), a method that improves generation quality by aligning training and inference through lightweight rollouts and contrastive supervision.

Contribution

The paper introduces SAR, a novel post-training technique combining Stagger-Scale Rollout and Contrastive Student-Forcing Loss to enhance autoregressive model training and generation quality.

Findings

SAR reduces FID by 5.2% on ImageNet 256 within 10 epochs

SAR improves generation quality with minimal computational overhead

The method is scalable and effective as a post-training enhancement

Abstract

Recent advances in autoregressive (AR) generative models have produced increasingly powerful systems for media synthesis. Among them, next-scale prediction has emerged as a popular paradigm, where models generate images in a coarse-to-fine manner. However, scale-wise AR models suffer from exposure bias, which undermines generation quality. We identify two primary causes of this issue: (1) train-test mismatch, where the model must rely on its own imperfect predictions during inference, and (2) imbalance in scale-wise learning difficulty, where certain scales exhibit disproportionately higher optimization complexity. Through a comprehensive analysis of training dynamics, we propose Self-Autoregressive Refinement (SAR) to address these limitations. SAR introduces a Stagger-Scale Rollout (SSR) mechanism that performs lightweight autoregressive rollouts to expose the model to its own intermediate predictions, thereby aligning train-test patterns, and a complementary Contrastive Student-Forcing Loss (CSFL) that provides adequate supervision for self-generated contexts to ensure stable training. Experimental results show that applying SAR to pretrained AR models consistently improves generation quality with minimal computational overhead. For instance, SAR yields a 5.2% FID reduction on FlexVAR-d16 trained on ImageNet 256 within 10 epochs (5 hours on 32xA100 GPUs). Given its efficiency, scalability, and effectiveness, we expect SAR to serve as a reliable post-training method for visual autoregressive generation.