Distillation of Discrete Diffusion by Exact Conditional Distribution Matching

TL;DR

This paper introduces a novel distillation method for discrete diffusion models that directly matches conditional distributions, significantly reducing inference costs while maintaining quality.

Contribution

It proposes a principled distillation approach based on exact conditional distribution matching, leveraging the Markov structure of the reverse process for efficient sampling.

Findings

Reduces the number of function evaluations needed for sampling.

Maintains high-quality generation with fewer steps.

Provides a theoretically grounded distillation framework.

Abstract

Discrete diffusion models (DDMs) are a powerful class of generative models for categorical data, but they typically require many function evaluations for a single sample, making inference expensive. Existing acceleration methods either rely on approximate simulators, such as $\tau$-leaping, or on distillation schemes that train new student models and auxiliary networks with proxy objectives. We propose a simple and principled distillation alternative based on \emph{conditional distribution matching}. Our key observation is that the reverse conditional distribution of clean data given a noisy state, $p_{0\mid t}(x_0 \mid x_t)$, admits a Markov decomposition through intermediate times and can be recovered from marginal density ratios and the known forward CTMC kernel. We exploit this structure to define distillation objectives that directly match conditional distributions between a pre-trained teacher and a low-NFE student, both for one-step and few-step samplers.