Scaling Laws For Diffusion Transformers

TL;DR

This paper establishes and validates scaling laws for diffusion transformers, enabling precise predictions of model performance, optimal size, and data needs across different compute budgets, thereby improving efficiency in content generation tasks.

Contribution

First to empirically confirm power-law scaling laws in diffusion transformers, linking compute, model size, and performance for better predictability.

Findings

Pretraining loss follows a power-law with compute.

Scaling laws enable accurate prediction of generation quality.

Trend of loss matches actual generation performance across datasets.

Abstract

Diffusion transformers (DiT) have already achieved appealing synthesis and scaling properties in content recreation, e.g., image and video generation. However, scaling laws of DiT are less explored, which usually offer precise predictions regarding optimal model size and data requirements given a specific compute budget. Therefore, experiments across a broad range of compute budgets, from 1e17 to 6e18 FLOPs are conducted to confirm the existence of scaling laws in DiT for the first time. Concretely, the loss of pretraining DiT also follows a power-law relationship with the involved compute. Based on the scaling law, we can not only determine the optimal model size and required data but also accurately predict the text-to-image generation loss given a model with 1B parameters and a compute budget of 1e21 FLOPs. Additionally, we also demonstrate that the trend of pre-training loss matches the generation performances (e.g., FID), even across various datasets, which complements the mapping from compute to synthesis quality and thus provides a predictable benchmark that assesses model performance and data quality at a reduced cost.