Glauber Generative Model: Discrete Diffusion Models via Binary Classification

TL;DR

The paper introduces Glauber Generative Model (GGM), a novel discrete diffusion approach using binary classification and heat bath dynamics, improving language and image generation tasks without dataset-specific tokenizers.

Contribution

GGM provides an exact reduction of discrete denoising to binary classification, offering a new framework that outperforms existing models in language and image generation.

Findings

Outperforms existing discrete diffusion models in language generation

Demonstrates strong image generation without dataset-specific tokenizers

Effective in zero-shot control tasks like infilling

Abstract

We introduce the Glauber Generative Model (GGM), a new class of discrete diffusion models, to obtain new samples from a distribution given samples from a discrete space. GGM deploys a discrete Markov chain called the heat bath dynamics (or the Glauber dynamics) to denoise a sequence of noisy tokens to a sample from a joint distribution of discrete tokens. Our novel conceptual framework provides an exact reduction of the task of learning the denoising Markov chain to solving a class of binary classification tasks. More specifically, the model learns to classify a given token in a noisy sequence as signal or noise. In contrast, prior works on discrete diffusion models either solve regression problems to learn importance ratios, or minimize loss functions given by variational approximations. We apply GGM to language modeling and image generation, where images are discretized using image tokenizers like VQGANs. We show that it outperforms existing discrete diffusion models in language generation, and demonstrates strong performance for image generation without using dataset-specific image tokenizers. We also show that our model is capable of performing well in zero-shot control settings like text and image infilling.