Forward-Learned Discrete Diffusion: Learning how to noise to denoise faster

TL;DR

The paper introduces Forward-Learned Discrete Diffusion (FLDD), a novel approach that learns the forward noising process to enable faster, high-quality discrete diffusion sampling with fewer steps.

Contribution

It proposes a non-Markovian, learnable forward process for discrete diffusion models, improving sampling efficiency and quality over traditional fixed processes.

Findings

FLDD achieves higher sample quality with fewer steps.

The approach outperforms conventional discrete diffusion models.

End-to-end training under the variational objective is effective.

Abstract

Discrete diffusion models are a powerful class of generative models with strong performance across many domains. For efficiency, however, discrete diffusion typically parameterizes the generative (reverse) process with factorized distributions, which makes it difficult for the model to learn the target process in a small number of steps and necessitates a long, computationally expensive sampling procedure. To reduce the gap between the target and model distributions and enable few-step generation, we propose Forward-Learned Discrete Diffusion (FLDD), which introduces discrete diffusion with a learnable forward (noising) process. Rather than fixing a Markovian forward chain, we adopt a non-Markovian formulation with learnable marginal and posterior distributions. This allows the generative process to remain factorized while matching the target defined by the noising process. We train all parameters end-to-end under the standard variational objective. Experiments on various benchmarks show that, for a given number of sampling steps, our approach produces a higher quality samples than conventional discrete diffusion models using the same reverse parameterization.