Exploring the Optimal Choice for Generative Processes in Diffusion Models: Ordinary vs Stochastic Differential Equations

TL;DR

This paper compares ODE-based and SDE-based diffusion models in generative tasks, analyzing their performance under different error scenarios and diffusion intensities through mathematical and empirical methods.

Contribution

It provides a theoretical analysis of when ODE or SDE models outperform each other based on error timing and diffusion strength, supported by numerical experiments.

Findings

ODE outperforms SDE with end-of-process perturbations at large diffusion.

SDE outperforms ODE with early perturbations, error exponentially decreases with diffusion.

Numerical validation on various datasets confirms theoretical insights.

Abstract

The diffusion model has shown remarkable success in computer vision, but it remains unclear whether the ODE-based probability flow or the SDE-based diffusion model is more superior and under what circumstances. Comparing the two is challenging due to dependencies on data distributions, score training, and other numerical issues. In this paper, we study the problem mathematically for two limiting scenarios: the zero diffusion (ODE) case and the large diffusion case. We first introduce a pulse-shape error to perturb the score function and analyze error accumulation of sampling quality, followed by a thorough analysis for generalization to arbitrary error. Our findings indicate that when the perturbation occurs at the end of the generative process, the ODE model outperforms the SDE model with a large diffusion coefficient. However, when the perturbation occurs earlier, the SDE model outperforms the ODE model, and we demonstrate that the error of sample generation due to such a pulse-shape perturbation is exponentially suppressed as the diffusion term's magnitude increases to infinity. Numerical validation of this phenomenon is provided using Gaussian, Gaussian mixture, and Swiss roll distribution, as well as realistic datasets like MNIST and CIFAR-10.