Interleaved Gibbs Diffusion: Generating Discrete-Continuous Data with Implicit Constraints

TL;DR

Interleaved Gibbs Diffusion (IGD) is a new generative framework for discrete-continuous data that improves modeling of dependencies and constraints, achieving state-of-the-art results across various challenging tasks.

Contribution

IGD introduces a flexible, Gibbs-sampling based diffusion model that handles implicit constraints and dependencies in discrete-continuous data without domain-specific biases.

Findings

Significant improvement in 3-SAT performance.

State-of-the-art results on molecule, layout, and tabular data generation.

Flexible integration of discrete and continuous denoisers.

Abstract

We introduce Interleaved Gibbs Diffusion (IGD), a novel generative modeling framework for discrete-continuous data, focusing on problems with important, implicit and unspecified constraints in the data. Most prior works on discrete and discrete-continuous diffusion assume a factorized denoising distribution, which can hinder the modeling of strong dependencies between random variables in such problems. We empirically demonstrate a significant improvement in 3-SAT performance out of the box by switching to a Gibbs-sampling style discrete diffusion model which does not assume factorizability. Motivated by this, we introduce IGD which generalizes discrete time Gibbs sampling type Markov chain for the case of discrete-continuous generation. IGD allows for seamless integration between discrete and continuous denoisers while theoretically guaranteeing exact reversal of a suitable forward process. Further, it provides flexibility in the choice of denoisers, allows conditional generation via state-space doubling and inference time refinement. Empirical evaluations on three challenging generation tasks - molecule structures, layouts and tabular data - demonstrate state-of-the-art performance. Notably, IGD achieves state-of-the-art results without relying on domain-specific inductive biases like equivariant diffusion or auxiliary losses. We explore a wide range of modeling, and interleaving strategies along with hyperparameters in each of these problems.