Is Your Conditional Diffusion Model Actually Denoising?

TL;DR

This paper investigates how conditioned diffusion models often deviate from their ideal denoising process, introducing Schedule Deviation to measure this phenomenon and analyzing its causes.

Contribution

The paper introduces Schedule Deviation, a new metric to quantify deviations from ideal denoising in conditioned diffusion models, and provides theoretical insights into its causes.

Findings

Deviations occur regardless of model size or training data.

Schedule Deviation effectively measures denoising deviations.

Inductive bias towards smoothness explains the deviation phenomenon.

Abstract

We study the inductive biases of diffusion models with a conditioning-variable, which have seen widespread application as both text-conditioned generative image models and observation-conditioned continuous control policies. We observe that when these models are queried conditionally, their generations consistently deviate from the idealized "denoising" process upon which diffusion models are formulated, inducing disagreement between popular sampling algorithms (e.g. DDPM, DDIM). We introduce Schedule Deviation, a rigorous measure which captures the rate of deviation from a standard denoising process, and provide a methodology to compute it. Crucially, we demonstrate that the deviation from an idealized denoising process occurs irrespective of the model capacity or amount of training data. We posit that this phenomenon occurs due to the difficulty of bridging distinct denoising flows across different parts of the conditioning space and show theoretically how such a phenomenon can arise through an inductive bias towards smoothness.