A Score-Based Density Formula, with Applications in Diffusion Generative Models

TL;DR

This paper establishes a theoretical density formula for diffusion processes, explaining why ELBO optimization effectively trains diffusion models like DDPMs, and offers new insights into score-matching and diffusion-based methods.

Contribution

It introduces a density formula linking target density and score functions, providing a theoretical basis for ELBO optimization in diffusion generative models.

Findings

The density formula connects target density with score functions.

ELBO optimization nearly matches the true objective for DDPMs.

Provides new insights into score-matching in GANs and diffusion classifiers.

Abstract

Score-based generative models (SGMs) have revolutionized the field of generative modeling, achieving unprecedented success in generating realistic and diverse content. Despite empirical advances, the theoretical basis for why optimizing the evidence lower bound (ELBO) on the log-likelihood is effective for training diffusion generative models, such as DDPMs, remains largely unexplored. In this paper, we address this question by establishing a density formula for a continuous-time diffusion process, which can be viewed as the continuous-time limit of the forward process in an SGM. This formula reveals the connection between the target density and the score function associated with each step of the forward process. Building on this, we demonstrate that the minimizer of the optimization objective for training DDPMs nearly coincides with that of the true objective, providing a theoretical foundation for optimizing DDPMs using the ELBO. Furthermore, we offer new insights into the role of score-matching regularization in training GANs, the use of ELBO in diffusion classifiers, and the recently proposed diffusion loss.