Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data

TL;DR

This paper introduces RADD, a simplified absorbing discrete diffusion model that models conditional data distributions, reduces computation, and achieves state-of-the-art results in zero-shot language modeling benchmarks.

Contribution

The paper reveals an analytic form for the concrete score in absorbing diffusion, proposes RADD for efficient sampling, and unifies diffusion with autoregressive models, advancing language modeling techniques.

Findings

RADD achieves state-of-the-art perplexity on 5 zero-shot language benchmarks.

RADD reduces function evaluations by caching outputs during sampling.

Theoretical unification of diffusion models with autoregressive models.

Abstract

Discrete diffusion models with absorbing processes have shown promise in language modeling. The key quantities to be estimated are the ratios between the marginal probabilities of two transitive states at all timesteps, called the concrete score. In this paper, we reveal that the concrete score in absorbing diffusion can be expressed as conditional probabilities of clean data, multiplied by a time-dependent scalar in an analytic form. Motivated by this finding, we propose reparameterized absorbing discrete diffusion (RADD), a dedicated diffusion model without time-condition that characterizes the time-independent conditional probabilities. Besides its simplicity, RADD can reduce the number of function evaluations (NFEs) by caching the output of the time-independent network when the noisy sample remains unchanged in a sampling interval, which enables sampling acceleration. Built upon the new perspective of conditional distributions, we further unify absorbing discrete diffusion and any-order autoregressive models (AO-ARMs), showing that the upper bound on the negative log-likelihood for the diffusion model can be interpreted as an expected negative log-likelihood for AO-ARMs. Further, our RADD models achieve SOTA performance among diffusion models on 5 zero-shot language modeling benchmarks (measured by perplexity) at the GPT-2 scale. Our code is available at https://github.com/ML-GSAI/RADD.