Data Warmup: Complexity-Aware Curricula for Efficient Diffusion Training

TL;DR

This paper introduces Data Warmup, a curriculum learning strategy that schedules training images from simple to complex to improve diffusion model training efficiency and quality.

Contribution

It proposes a complexity-aware curriculum based on semantic metrics, enhancing diffusion training speed and performance without altering the model or loss.

Findings

Improves IS by up to 6.11 and FID by up to 3.41 on ImageNet 256x256.

Achieves baseline quality significantly earlier in training.

Simple-to-complex curriculum outperforms reversing the order.

Abstract

A key inefficiency in diffusion training occurs when a randomly initialized network, lacking visual priors, encounters gradients from the full complexity spectrum--most of which it lacks the capacity to resolve. We propose Data Warmup, a curriculum strategy that schedules training images from simple to complex without modifying the model or loss. Each image is scored offline by a semantic-aware complexity metric combining foreground dominance (how much of the image salient objects occupy) and foreground typicality (how closely the salient content matches learned visual prototypes). A temperature-controlled sampler then prioritizes low-complexity images early and anneals toward uniform sampling. On ImageNet 256x256 with SiT backbones (S/2 to XL/2), Data Warmup improves IS by up to 6.11 and FID by up to 3.41, reaching baseline quality tens of thousands of iterations earlier. Reversing the curriculum (exposing hard images first) degrades performance below the uniform baseline, confirming that the simple-to-complex ordering itself drives the gains. The method combines with orthogonal accelerators such as REPA and requires only ~10 minutes of one-time preprocessing with zero per-iteration overhead.