Diffusion Beats Autoregressive in Data-Constrained Settings

TL;DR

This paper demonstrates that diffusion-based language models outperform autoregressive models in data-constrained settings by effectively utilizing limited data through implicit data augmentation, especially when compute resources are abundant.

Contribution

It introduces a systematic study of masked diffusion models in low-data scenarios, deriving new scaling laws and explaining their advantages over AR models.

Findings

Diffusion models outperform AR models with limited data and ample compute.

New scaling laws for diffusion models are established.

A closed-form expression for the compute threshold where diffusion surpasses AR.

Abstract

Autoregressive (AR) models have long dominated the landscape of large language models, driving progress across a wide range of tasks. Recently, diffusion-based language models have emerged as a promising alternative, though their advantages over AR models remain underexplored. In this paper, we systematically study masked diffusion models in data-constrained settings where training involves repeated passes over limited data and find that they significantly outperform AR models when compute is abundant but data is scarce. Diffusion models make better use of repeated data, achieving lower validation loss and superior downstream performance. We find new scaling laws for diffusion models and derive a closed-form expression for the critical compute threshold at which diffusion begins to outperform AR. Finally, we explain why diffusion models excel in this regime: their randomized masking objective implicitly trains over a rich distribution of token orderings, acting as an implicit data augmentation that AR's fixed left-to-right factorization lacks. Our results suggest that when data, not compute, is the bottleneck, diffusion models offer a compelling alternative to the standard AR paradigm. Our code is available at: https://diffusion-scaling.github.io.