Spectral Analysis of Diffusion Models with Application to Schedule Design

TL;DR

This paper introduces a spectral analysis framework for diffusion models, providing a theoretical basis for designing noise schedules aligned with data spectral properties, enhancing understanding of the inference process.

Contribution

It offers a novel frequency response perspective of diffusion model inference, enabling principled noise schedule design based on spectral data characteristics.

Findings

Spectral transfer function characterizes diffusion inference dynamics.

Data-dependent noise schedules improve synthesis quality.

Theoretical insights justify existing heuristic scheduling methods.

Abstract

Diffusion models (DMs) have emerged as powerful tools for modeling complex data distributions and generating realistic new samples. Over the years, advanced architectures and sampling methods have been developed to make these models practically usable. However, certain synthesis process decisions still rely on heuristics without a solid theoretical foundation. In our work, we offer a novel analysis of the DM's inference process, introducing a comprehensive frequency response perspective. Specifically, by relying on Gaussianity assumption, we present the inference process as a closed-form spectral transfer function, capturing how the generated signal evolves in response to the initial noise. We demonstrate how the proposed analysis can be leveraged to design a noise schedule that aligns effectively with the characteristics of the data. The spectral perspective also provides insights into the underlying dynamics and sheds light on the relationship between spectral properties and noise schedule structure. Our results lead to scheduling curves that are dependent on the spectral content of the data, offering a theoretical justification for some of the heuristics taken by practitioners.