Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms

TL;DR

This paper introduces high-order numerical inference schemes for discrete diffusion models, significantly improving efficiency and sample quality across various tasks by enabling larger step sizes and reducing errors.

Contribution

It develops the first high-order inference algorithms for discrete diffusion models, providing theoretical analysis and demonstrating superior performance over existing methods.

Findings

Second-order accuracy of the $ heta$-Trapezoidal method established.

Achieves better sample quality with the same computational cost.

Consistent improvements across multiple models and tasks.

Abstract

Discrete diffusion models have emerged as a powerful generative modeling framework for discrete data with successful applications spanning from text generation to image synthesis. However, their deployment faces challenges due to the high dimensionality of the state space, necessitating the development of efficient inference algorithms. Current inference approaches mainly fall into two categories: exact simulation and approximate methods such as $\tau$-leaping. While exact methods suffer from unpredictable inference time and redundant function evaluations, $\tau$-leaping is limited by its first-order accuracy. In this work, we advance the latter category by tailoring the first extension of high-order numerical inference schemes to discrete diffusion models, enabling larger step sizes while reducing error. We rigorously analyze the proposed schemes and establish the second-order accuracy of the $\theta$-Trapezoidal method in KL divergence. Empirical evaluations on GSM8K-level math-reasoning, GPT-2-level text, and ImageNet-level image generation tasks demonstrate that our method achieves superior sample quality compared to existing approaches under equivalent computational constraints, with consistent performance gains across models ranging from 200M to 8B. Our code is available at https://github.com/yuchen-zhu-zyc/DiscreteFastSolver.