Debiasing Guidance for Discrete Diffusion with Sequential Monte Carlo

TL;DR

This paper introduces a Sequential Monte Carlo method for discrete diffusion models that achieves unbiased sampling from targeted distributions, improving control in text and image generation tasks.

Contribution

The paper presents a novel SMC algorithm that unbiasedly samples from targeted distributions in discrete diffusion models, addressing limitations of existing guidance methods.

Findings

Unbiased sampling from target distributions achieved.

Enhanced control in text and image generation.

Maintains low perplexity compared to existing guidance methods.

Abstract

Discrete diffusion models are a class of generative models that produce samples from an approximated data distribution within a discrete state space. Often, there is a need to target specific regions of the data distribution. Current guidance methods aim to sample from a distribution with mass proportional to $p_0(x_0) p(\zeta|x_0)^\alpha$ but fail to achieve this in practice. We introduce a Sequential Monte Carlo algorithm that generates unbiasedly from this target distribution, utilising the learnt unconditional and guided process. We validate our approach on low-dimensional distributions, controlled images and text generations. For text generation, our method provides strong control while maintaining low perplexity compared to guidance-based approaches.