Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models

TL;DR

This paper proposes a model scheduling technique for masked diffusion language models that reduces computational cost by replacing the full model with a smaller one at certain denoising steps, mainly in the early and late stages.

Contribution

It introduces a simple, architecture-agnostic scheduling method that accelerates MDLM sampling with minimal impact on quality, supported by analysis and extensive experiments.

Findings

Up to 17% FLOPs reduction with modest perplexity increase

Early and late denoising steps are more robust to model replacement

Middle steps are most sensitive to model replacement

Abstract

Recent advances in masked diffusion language models (MDLMs) narrow the quality gap to autoregressive LMs, but their sampling remains expensive because generation requires many full-sequence denoising passes with a large Transformer and, unlike autoregressive decoding, cannot benefit from KV caching. In this work, we exploit the flexibility of the diffusion framework and study model scheduling, where a smaller MDLM replaces the full model at a subset of denoising steps. Across models trained on OpenWebText and LM1B, we show that early and late denoising steps are substantially more robust to such replacement than middle steps, enabling up to a 17% reduction in FLOPs with only modest degradation in generative perplexity under both unconditional and prefix-conditional generation, while preserving sample diversity. We support these findings with a step-importance analysis based on loss and KL divergence between small and large models across timesteps, as well as an exhaustive search over coarse step segments, both of which identify the middle of the diffusion trajectory as most sensitive consistently across datasets. Our results suggest that simple, architecture-agnostic scheduling rules can significantly accelerate MDLM sampling while largely preserving generation quality.