Scaling Behavior of Discrete Diffusion Language Models

TL;DR

This paper investigates the scaling laws of discrete diffusion language models, revealing how different noise types affect their efficiency and demonstrating the successful training of a 10-billion-parameter uniform diffusion model.

Contribution

It provides the first comprehensive analysis of DLM scaling behavior across noise types and introduces a large-scale uniform diffusion model, highlighting its potential advantages.

Findings

Scaling behavior varies significantly with noise type.

Uniform diffusion models are more data-efficient in compute-bound regimes.

Trained the largest known uniform diffusion model with 10B parameters.

Abstract

Modern LLM pre-training consumes vast amounts of compute and training data, making the scaling behavior, or scaling laws, of different models a key distinguishing factor. Discrete diffusion language models (DLMs) have been proposed as an alternative to autoregressive language models (ALMs). However, their scaling behavior has not yet been fully explored, with prior work suggesting that they require more data and compute to match the performance of ALMs. We study the scaling behavior of DLMs on different noise types by smoothly interpolating between masked and uniform diffusion while paying close attention to crucial hyperparameters such as batch size and learning rate. Our experiments reveal that the scaling behavior of DLMs strongly depends on the noise type and is considerably different from ALMs. While all noise types converge to similar loss values in compute-bound scaling, we find that uniform diffusion requires more parameters and less data for compute-efficient training compared to masked diffusion, making them a promising candidate in data-bound settings. We scale our uniform diffusion model up to 10B parameters trained for $10^{22}$ FLOPs, confirming the predicted scaling behavior and making it the largest publicly known uniform diffusion model to date.