GUD: Generation with Unified Diffusion

TL;DR

This paper introduces GUD, a unified diffusion framework inspired by physics concepts, allowing flexible design choices in representation, prior, and noise scheduling to improve generative modeling.

Contribution

It presents a novel unified framework for diffusion models that integrates various representations, priors, and schedules, enabling smoother interpolation between diffusion and autoregressive models.

Findings

Enhanced design flexibility for diffusion models.

Bridging diffusion and autoregressive approaches.

Potential for more efficient training and generation.

Abstract

Diffusion generative models transform noise into data by inverting a process that progressively adds noise to data samples. Inspired by concepts from the renormalization group in physics, which analyzes systems across different scales, we revisit diffusion models by exploring three key design aspects: 1) the choice of representation in which the diffusion process operates (e.g. pixel-, PCA-, Fourier-, or wavelet-basis), 2) the prior distribution that data is transformed into during diffusion (e.g. Gaussian with covariance $\Sigma$), and 3) the scheduling of noise levels applied separately to different parts of the data, captured by a component-wise noise schedule. Incorporating the flexibility in these choices, we develop a unified framework for diffusion generative models with greatly enhanced design freedom. In particular, we introduce soft-conditioning models that smoothly interpolate between standard diffusion models and autoregressive models (in any basis), conceptually bridging these two approaches. Our framework opens up a wide design space which may lead to more efficient training and data generation, and paves the way to novel architectures integrating different generative approaches and generation tasks.