Discrete vs. Continuous Trade-offs for Generative Models

TL;DR

This paper analyzes the theoretical foundations and performance trade-offs of denoising diffusion and score-based generative models, focusing on stochastic processes, noise propagation, and bounds on data distribution approximation.

Contribution

It provides a novel theoretical analysis of the performance bounds and noise propagation in diffusion-based generative models using information theory.

Findings

Score estimation errors affect data generation quality

Performance bounds are derived using information-theoretic inequalities

Trade-offs between discrete and continuous diffusion processes are characterized

Abstract

This work explores the theoretical and practical foundations of denoising diffusion probabilistic models (DDPMs) and score-based generative models, which leverage stochastic processes and Brownian motion to model complex data distributions. These models employ forward and reverse diffusion processes defined through stochastic differential equations (SDEs) to iteratively add and remove noise, enabling high-quality data generation. By analyzing the performance bounds of these models, we demonstrate how score estimation errors propagate through the reverse process and bound the total variation distance using discrete Girsanov transformations, Pinsker's inequality, and the data processing inequality (DPI) for an information theoretic lens.