Discrete Diffusion with Sample-Efficient Estimators for Conditionals

TL;DR

This paper introduces a discrete diffusion framework that uses a sample-efficient estimator for single-site conditionals, improving generative modeling over discrete spaces with better performance than existing methods.

Contribution

It presents a novel approach that treats single-site conditional probabilities as fundamental, utilizing NeurISE for efficient estimation within a diffusion process for discrete data.

Findings

Outperforms existing ratio-based methods on binary datasets

Achieves better total variation, cross-correlations, and kernel density metrics

Demonstrates effectiveness on synthetic and real-world data, including quantum and scientific datasets

Abstract

We study a discrete denoising diffusion framework that integrates a sample-efficient estimator of single-site conditionals with round-robin noising and denoising dynamics for generative modeling over discrete state spaces. Rather than approximating a discrete analog of a score function, our formulation treats single-site conditional probabilities as the fundamental objects that parameterize the reverse diffusion process. We employ a sample-efficient method known as Neural Interaction Screening Estimator (NeurISE) to estimate these conditionals in the diffusion dynamics. Controlled experiments on synthetic Ising models, MNIST, and scientific data sets produced by a D-Wave quantum annealer, synthetic Potts model and one-dimensional quantum systems demonstrate the proposed approach. On the binary data sets, these experiments demonstrate that the proposed approach outperforms popular existing methods including ratio-based approaches, achieving improved performance in total variation, cross-correlations, and kernel density estimation metrics.