Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison

TL;DR

This study provides a controlled empirical comparison between autoregressive and masked diffusion language models, revealing differences in training speed, convergence, diversity, and fluency under identical conditions.

Contribution

It is the first to compare AR and MDLM models under identical training data, compute, and hardware, isolating the generation paradigm as the key variable.

Findings

AR converges faster and overfits earlier

MDLM produces more diverse outputs

AR outputs are more fluent but less diverse

Abstract

We present a controlled empirical comparison between autoregressive (AR) and masked diffusion (MDLM) language models. Both models are trained on identical data (50M tokens from TinyStories), identical compute budget (20,000 steps, batch size 32, sequence length 512), and identical hardware (NVIDIA H100 80GB), isolating the generation paradigm as the sole variable. We report three findings. First, both paradigms achieve comparable training throughput (~50K tokens/second), with MDLM requiring only 4.7% more wall-clock time. Second, AR converges faster and begins overfitting by step 14,000, while MDLM converges more slowly and is still improving at step 20,000, suggesting different compute-optimal training regimes. Third, quantitative diversity analysis over 1,000 generated samples reveals a structural diversity-fluency trade-off: AR produces fluent but repetitive outputs (99.8% begin with the same word), while MDLM generates more diverse narratives (93.4% unique 5-word openings, higher Distinct-n, lower Self-BLEU), at the cost of occasional grammatical inconsistencies. All code, trained checkpoints, and data pipelines are released for reproducibility.