Operator-Informed Score Matching for Markov Diffusion Models

TL;DR

This paper introduces an operator-informed score matching method that leverages the spectral properties of Markov noising processes to improve score estimation for diffusion models, enhancing sample generation especially in low-dimensional cases.

Contribution

It proposes a novel score matching approach using spectral decomposition of Markov operators, providing a unified framework for low- and high-dimensional diffusion modeling.

Findings

Effective score estimation for low-dimensional distributions

Improved neural score estimators in high-dimensional settings

Theoretical insights into Markov noising processes

Abstract

Diffusion models are typically trained using score matching, a learning objective agnostic to the underlying noising process that guides the model. This paper argues that Markov noising processes enjoy an advantage over alternatives, as the Markov operators that govern the noising process are well-understood. Specifically, by leveraging the spectral decomposition of the infinitesimal generator of the Markov noising process, we obtain parametric estimates of the score functions simultaneously for all marginal distributions, using only sample averages with respect to the data distribution. The resulting operator-informed score matching provides both a standalone approach to sample generation for low-dimensional distributions, as well as a recipe for better informed neural score estimators in high-dimensional settings.