Masked Auto-Regressive Variational Acceleration: Fast Inference Makes Practical Reinforcement Learning

TL;DR

This paper introduces MARVAL, a distillation framework that accelerates masked auto-regressive diffusion models, making reinforcement learning practical by enabling fast inference without sacrificing sample quality.

Contribution

The paper proposes a novel score-based variational objective for distilling diffusion models into a single step and develops an efficient RL framework for masked auto-regressive models.

Findings

MARVAL achieves over 30x speedup on ImageNet 256x256.

MARVAL-Huge attains an FID of 2.00 on ImageNet 256x256.

MARVAL-RL improves CLIP and image-reward scores on ImageNet datasets.

Abstract

Masked auto-regressive diffusion models (MAR) benefit from the expressive modeling ability of diffusion models and the flexibility of masked auto-regressive ordering. However, vanilla MAR suffers from slow inference due to its hierarchical inference mechanism: an outer AR unmasking loop and an inner diffusion denoising chain. Such decoupled structure not only harm the generation efficiency but also hinder the practical use of MAR for reinforcement learning (RL), an increasingly critical paradigm for generative model post-training.To address this fundamental issue, we introduce MARVAL (Masked Auto-regressive Variational Acceleration), a distillation-based framework that compresses the diffusion chain into a single AR generation step while preserving the flexible auto-regressive unmasking order. Such a distillation with MARVAL not only yields substantial inference acceleration but, crucially, makes RL post-training with verifiable rewards practical, resulting in scalable yet human-preferred fast generative models. Our contributions are twofold: (1) a novel score-based variational objective for distilling masked auto-regressive diffusion models into a single generation step without sacrificing sample quality; and (2) an efficient RL framework for masked auto-regressive models via MARVAL-RL. On ImageNet 256*256, MARVAL-Huge achieves an FID of 2.00 with more than 30 times speedup compared with MAR-diffusion, and MARVAL-RL yields consistent improvements in CLIP and image-reward scores on ImageNet datasets with entity names. In conclusion, MARVAL demonstrates the first practical path to distillation and RL of masked auto-regressive diffusion models, enabling fast sampling and better preference alignments.