Likelihood Matching for Diffusion Models

TL;DR

This paper introduces a Likelihood Matching method for training diffusion models by approximating transition densities with Gaussian distributions, ensuring consistent likelihood estimation and providing convergence guarantees.

Contribution

It presents a novel likelihood matching framework for diffusion models, including a quasi-likelihood approximation, a stochastic sampler, and theoretical convergence analysis.

Findings

Effective likelihood matching improves diffusion model training

The proposed sampler converges with quantifiable error bounds

Empirical results validate theoretical guarantees

Abstract

We propose a Likelihood Matching approach for training diffusion models by first establishing an equivalence between the likelihood of the target data distribution and a likelihood along the sample path of the reverse diffusion. To efficiently compute the reverse sample likelihood, a quasi-likelihood is considered to approximate each reverse transition density by a Gaussian distribution with matched conditional mean and covariance, respectively. The score and Hessian functions for the diffusion generation are estimated by maximizing the quasi-likelihood, ensuring a consistent matching of both the first two transitional moments between every two time points. A stochastic sampler is introduced to facilitate computation that leverages both the estimated score and Hessian information. We establish consistency of the quasi-maximum likelihood estimation, and provide non-asymptotic convergence guarantees for the proposed sampler, quantifying the rates of the approximation errors due to the score and Hessian estimation, dimensionality, and the number of diffusion steps. Empirical and simulation evaluations demonstrate the effectiveness of the proposed Likelihood Matching and validate the theoretical results.