Informed Correctors for Discrete Diffusion Models

TL;DR

This paper introduces an informed predictor-corrector sampling scheme for discrete diffusion models, significantly improving sampling efficiency and quality in discrete generative tasks.

Contribution

It proposes a novel predictor-corrector method with model-informed correction, architectural enhancements, and a tailored training objective to improve discrete diffusion sampling.

Findings

Superior sample quality on text8 and ImageNet datasets

Fewer errors and better FID scores with the proposed method

Effective alleviation of sampling failure modes

Abstract

Discrete diffusion has emerged as a powerful framework for generative modeling in discrete domains, yet efficiently sampling from these models remains challenging. Existing sampling strategies often struggle to balance computation and sample quality when the number of sampling steps is reduced, even when the model has learned the data distribution well. To address these limitations, we propose a predictor-corrector sampling scheme where the corrector is informed by the diffusion model to more reliably counter the accumulating approximation errors. To further enhance the effectiveness of our informed corrector, we introduce complementary architectural modifications based on hollow transformers and a simple tailored training objective that leverages more training signal. We use a synthetic example to illustrate the failure modes of existing samplers and show how informed correctors alleviate these problems. On the text8 and tokenized ImageNet 256x256 datasets, our informed corrector consistently produces superior samples with fewer errors or improved FID scores for discrete diffusion models. These results underscore the potential of informed correctors for fast and high-fidelity generation using discrete diffusion. Our code is available at https://github.com/lindermanlab/informed-correctors.