The Diffusion Duality

TL;DR

This paper introduces Duo, a diffusion model that leverages Gaussian diffusion techniques to enhance training speed and sampling efficiency, surpassing autoregressive models in some benchmarks for text generation.

Contribution

The paper proposes a novel method, Duo, which transfers Gaussian diffusion techniques to discrete diffusion models, improving training speed and enabling fast sampling in language models.

Findings

Models with curriculum learning outperform autoregressive models in zero-shot perplexity.

Discreet Consistency Distillation accelerates sampling by two orders of magnitude.

Duo achieves competitive results on multiple benchmarks.

Abstract

Uniform-state discrete diffusion models hold the promise of fast text generation due to their inherent ability to self-correct. However, they are typically outperformed by autoregressive models and masked diffusion models. In this work, we narrow this performance gap by leveraging a key insight: Uniform-state diffusion processes naturally emerge from an underlying Gaussian diffusion. Our method, Duo, transfers powerful techniques from Gaussian diffusion to improve both training and sampling. First, we introduce a curriculum learning strategy guided by the Gaussian process, doubling training speed by reducing variance. Models trained with curriculum learning surpass autoregressive models in zero-shot perplexity on 3 of 7 benchmarks. Second, we present Discrete Consistency Distillation, which adapts consistency distillation from the continuous to the discrete setting. This algorithm unlocks few-step generation in diffusion language models by accelerating sampling by two orders of magnitude. We provide the code, model checkpoints, and video tutorials on the project page: http://s-sahoo.github.io/duo