Using Ornstein-Uhlenbeck Process to understand Denoising Diffusion Probabilistic Model and its Noise Schedules

TL;DR

This paper reveals that Denoising Diffusion Probabilistic Models can be understood as Ornstein-Uhlenbeck processes, providing a new theoretical foundation and connecting noise scheduling to classical stochastic process design.

Contribution

It establishes a formal equivalence between DDPM and the OU process, offering a new perspective and principled methods for designing noise schedules.

Findings

Fisher-information-based schedule matches the cosine schedule.

DDPM can be modeled as a time-homogeneous OU process.

New insights into noise schedule design for diffusion models.

Abstract

The aim of this short note is to show that Denoising Diffusion Probabilistic Model DDPM, a non-homogeneous discrete-time Markov process, can be represented by a time-homogeneous continuous-time Markov process observed at non-uniformly sampled discrete times. Surprisingly, this continuous-time Markov process is the well-known and well-studied Ornstein-Ohlenbeck (OU) process, which was developed in 1930's for studying Brownian particles in Harmonic potentials. We establish the formal equivalence between DDPM and the OU process using its analytical solution. We further demonstrate that the design problem of the noise scheduler for non-homogeneous DDPM is equivalent to designing observation times for the OU process. We present several heuristic designs for observation times based on principled quantities such as auto-variance and Fisher Information and connect them to ad hoc noise schedules for DDPM. Interestingly, we show that the Fisher-Information-motivated schedule corresponds exactly the cosine schedule, which was developed without any theoretical foundation but is the current state-of-the-art noise schedule.