Generative diffusion models from a PDE perspective

TL;DR

This paper provides a PDE-based analysis of diffusion models, revealing their limitations in regularization and generalization, and linking discrete and continuous approaches through explicit solutions of reverse dynamics.

Contribution

It offers a PDE perspective on diffusion models, derives explicit solutions for reverse processes, and clarifies the connection between different generative diffusion methods.

Findings

Reverse distribution support is within the original distribution.

Diffusion methods do not inherently regularize the original distribution.

Exact solutions lead to overfitting to original data points.

Abstract

Diffusion models have become the de facto framework for generating new datasets. The core of these models lies in the ability to reverse a diffusion process in time. The goal of this manuscript is to explain, from a PDE perspective, how this method works and how to derive the PDE governing the reverse dynamics as well as to study its solution analytically. By linking forward and reverse dynamics, we show that the reverse process's distribution has its support contained within the original distribution. Consequently, diffusion methods, in their analytical formulation, do not inherently regularize the original distribution, and thus, there is no generalization principle. This raises a question: where does generalization arise, given that in practice it does occur? Moreover, we derive an explicit solution to the reverse process's SDE under the assumption that the starting point of the forward process is fixed. This provides a new derivation that links two popular approaches to generative diffusion models: stable diffusion (discrete dynamics) and the score-based approach (continuous dynamics). Finally, we explore the case where the original distribution consists of a finite set of data points. In this scenario, the reverse dynamics are explicit (i.e., the loss function has a clear minimizer), and solving the dynamics fails to generate new samples: the dynamics converge to the original samples. In a sense, solving the minimization problem exactly is "too good for its own good" (i.e., an overfitting regime).