Tweedie's Formulae and Diffusion Generative Models Beyond Gaussian

TL;DR

This paper extends Tweedie's formulae to non-Gaussian diffusion processes like GBM, BESQ, and CIR, enabling new generative modeling and empirical Bayes applications beyond Gaussian assumptions.

Contribution

It introduces novel Tweedie's formulae for non-Gaussian processes and demonstrates their application in image, financial data generation, and empirical Bayes estimation.

Findings

Non-Gaussian diffusion models show promising results in image and financial data generation.

Extended Tweedie's formulae facilitate denoising score matching for non-Gaussian processes.

Experimental results highlight the potential of non-Gaussian models in various applications.

Abstract

Diffusion models have achieved remarkable success in generating samples from unknown data distributions. Most popular stochastic differential equation-based diffusion models perturb the target distribution by adding Gaussian noise, transforming it into a simple prior, and then use denoising score matching, a consequence of Tweedie's formula, to learn the score function and generate clean samples from noise. However, non-Gaussian diffusion models with state-dependent diffusion coefficient have been largely underexplored, as have the corresponding Tweedie's formulae. In this work, we extend Tweedie's formula to important non-Gaussian processes, including geometric Brownian motion (GBM), squared Bessel (BESQ) processes, and Cox-Ingersoll-Ross (CIR) processes, thereby yielding the corresponding denoising score-matching objectives. We then apply the derived formulae to image and financial time series generation using GBM- and CIR-based diffusion models, and to empirical Bayes estimation under the BESQ setting. The reported experimental results demonstrate the potential of non-Gaussian models.