Information-Guided Noise Allocation for Efficient Diffusion Training

TL;DR

This paper introduces InfoNoise, an information-theoretic, data-adaptive noise schedule for diffusion models that improves training efficiency and quality across various datasets by replacing heuristic schedules with entropy-based noise sampling.

Contribution

The paper proposes a novel, theoretically grounded noise scheduling method, InfoNoise, which adapts to data using entropy reduction rates, outperforming traditional heuristic schedules.

Findings

InfoNoise matches or surpasses tuned schedules on natural images.

It achieves up to 1.4x faster training on CIFAR-10.

On discrete datasets, it reduces training steps by up to 3x.

Abstract

Training diffusion models typically relies on manually tuned noise schedules, which can waste computation on weakly informative noise regions and limit transfer across datasets, resolutions, and representations. We revisit noise schedule allocation through an information-theoretic lens and propose the conditional entropy rate of the forward process as a theoretically grounded, data-dependent diagnostic for identifying suboptimal noise-level allocation in existing schedules. Based on these insight, we introduce InfoNoise, a principled data-adaptive training noise schedule that replaces heuristic schedule design with an information-guided noise sampling distribution derived from entropy-reduction rates estimated from denoising losses already computed during training. Across natural-image benchmarks, InfoNoise matches or surpasses tuned EDM-style schedules, in some cases with a substantial training speedup (about $1.4\times$ on CIFAR-10). On discrete datasets, where standard image-tuned schedules exhibit significant mismatch, it reaches superior quality in up to $3\times$ fewer training steps. Overall, InfoNoise makes noise scheduling data-adaptive, reducing the need for per-dataset schedule design as diffusion models expand across domains.