Preserving Spectral Structure and Statistics in Diffusion Models

TL;DR

This paper introduces PreSS, a spectral space-based diffusion model that preserves spectral structure and statistics, leading to more efficient, diverse, and high-quality image generation compared to traditional pixel-based diffusion models.

Contribution

PreSS proposes a novel spectral space process that converges to an informative Gaussian prior, improving efficiency and diversity in diffusion-based image generation.

Findings

Reduces computational complexity significantly.

Enhances visual diversity and image quality.

Results on CIFAR-10 and CelebA datasets show improved performance.

Abstract

Standard diffusion models (DMs) rely on the total destruction of data into non-informative white noise, forcing the backward process to denoise from a fully unstructured noise state. While ensuring diversity, this results in a cumbersome and computationally intensive image generation task. We address this challenge by proposing new forward and backward process within a mathematically tractable spectral space. Unlike pixel-based DMs, our forward process converges towards an informative Gaussian prior N(mu_hat,Sigma_hat) rather than white noise. Our method, termed Preserving Spectral Structure and Statistics (PreSS) in diffusion models, guides spectral components toward this informative prior while ensuring that corresponding structural signals remain intact at terminal time. This provides a principled starting point for the backward process, enabling high-quality image reconstruction that builds upon preserved spectral structure while maintaining high generative diversity. Experimental results on CIFAR-10, CelebA and CelebA-HQ demonstrate significant reductions in computational complexity, improved visual diversity, less drift, and a smoother diffusion process compared to pixel-based DMs.