When Diffusion Model Can Ignore Dimension: An Entropy-Based Theory

TL;DR

This paper introduces an entropy-based theory explaining why diffusion models are efficient in high-dimensional data, showing that discretization error depends on latent entropy rather than ambient dimension.

Contribution

It provides a novel information-theoretic analysis demonstrating that diffusion sampler efficiency hinges on latent entropy, not ambient dimension, for Gaussian mixtures and discrete distributions.

Findings

Discretization error is controlled by the Shannon entropy of the latent mixture.

Leading step complexity scales linearly with latent entropy.

Results extend to discrete distributions, depending on target entropy.

Abstract

Diffusion models perform remarkably well on high-dimensional data such as images, often using only a modest number of reverse-time steps. Despite this practical success, existing convergence theory does not fully explain why such samplers remain efficient in high dimensions. Many prior KL guarantees bound the discretization error in terms of the ambient dimension, while other improved results replace this dependence using intrinsic-dimensional or geometric structure assumptions. In this work, we develop an alternative information-theoretic perspective on diffusion sampler convergence. We prove that, for Gaussian mixture targets, the discretization error is controlled by the Shannon entropy of the latent mixture component rather than by the ambient dimension. Consequently, the leading step complexity scales linearly with latent entropy and depends only logarithmically on the second moment of the data. Our analysis also extends to discrete target distributions, where the relevant complexity is the entropy of the target rather than the dimension of the embedding space. These results suggest that diffusion sampling can remain efficient in high-dimensional spaces when the data distribution admits a compact latent representation, as is widely believed to be the case for natural images.