Closing the ODE-SDE gap in score-based diffusion models through the Fokker-Planck equation

TL;DR

This paper analyzes the differences between ODE and SDE dynamics in score-based diffusion models, linking them to Fokker-Planck equations, and proposes a regularization method to close the distribution gap.

Contribution

It provides a theoretical analysis of ODE and SDE dynamics in diffusion models, introduces bounds on their distribution differences, and proposes a regularization to improve ODE sample quality.

Findings

Significant differences exist between ODE and SDE distributions in diffusion models.

Adding Fokker-Planck residual as regularization reduces the ODE-SDE distribution gap.

Regularization improves ODE-based sample quality but may degrade SDE samples.

Abstract

Score-based diffusion models have emerged as one of the most promising frameworks for deep generative modelling, due to their state-of-the art performance in many generation tasks while relying on mathematical foundations such as stochastic differential equations (SDEs) and ordinary differential equations (ODEs). Empirically, it has been reported that ODE based samples are inferior to SDE based samples. In this paper we rigorously describe the range of dynamics and approximations that arise when training score-based diffusion models, including the true SDE dynamics, the neural approximations, the various approximate particle dynamics that result, as well as their associated Fokker--Planck equations and the neural network approximations of these Fokker--Planck equations. We systematically analyse the difference between the ODE and SDE dynamics of score-based diffusion models, and link it to an associated Fokker--Planck equation. We derive a theoretical upper bound on the Wasserstein 2-distance between the ODE- and SDE-induced distributions in terms of a Fokker--Planck residual. We also show numerically that conventional score-based diffusion models can exhibit significant differences between ODE- and SDE-induced distributions which we demonstrate using explicit comparisons. Moreover, we show numerically that reducing the Fokker--Planck residual by adding it as an additional regularisation term leads to closing the gap between ODE- and SDE-induced distributions. Our experiments suggest that this regularisation can improve the distribution generated by the ODE, however that this can come at the cost of degraded SDE sample quality.