Blackout Diffusion: Generative Diffusion Models in Discrete-State Spaces

TL;DR

This paper introduces a theoretical framework for discrete-state diffusion models, exemplified by Blackout Diffusion, enabling sample generation from empty states and expanding diffusion modeling beyond Gaussian assumptions.

Contribution

It develops an exact theoretical formulation for discrete-state Markov processes in diffusion, including reverse processes, and introduces Blackout Diffusion for discrete data.

Findings

Successful implementation on CIFAR-10, Binarized MNIST, and CelebA datasets.

Demonstrates feasibility of discrete-state diffusion without variational approximations.

Provides insights into interpreting diffusion models in discrete spaces.

Abstract

Typical generative diffusion models rely on a Gaussian diffusion process for training the backward transformations, which can then be used to generate samples from Gaussian noise. However, real world data often takes place in discrete-state spaces, including many scientific applications. Here, we develop a theoretical formulation for arbitrary discrete-state Markov processes in the forward diffusion process using exact (as opposed to variational) analysis. We relate the theory to the existing continuous-state Gaussian diffusion as well as other approaches to discrete diffusion, and identify the corresponding reverse-time stochastic process and score function in the continuous-time setting, and the reverse-time mapping in the discrete-time setting. As an example of this framework, we introduce ``Blackout Diffusion'', which learns to produce samples from an empty image instead of from noise. Numerical experiments on the CIFAR-10, Binarized MNIST, and CelebA datasets confirm the feasibility of our approach. Generalizing from specific (Gaussian) forward processes to discrete-state processes without a variational approximation sheds light on how to interpret diffusion models, which we discuss.