Generalized Discrete Diffusion with Self-Correction

TL;DR

This paper introduces SCDD, a discrete diffusion model with explicit state transitions and learned self-correction, improving parallel decoding efficiency while maintaining high-quality generation.

Contribution

It reformulates self-correction in discrete diffusion models with explicit states, simplifying training and enhancing decoding efficiency.

Findings

Enables more efficient parallel decoding.

Preserves generation quality comparable to prior methods.

Simplifies training by removing redundant steps.

Abstract

Self-correction is an effective technique for maintaining parallel sampling in discrete diffusion models with minimal performance degradation. Prior work has explored self-correction at inference time or during post-training; however, such approaches often suffer from limited generalization and may impair reasoning performance. GIDD pioneers pretraining-based self-correction via a multi-step BERT-style uniform-absorbing objective. However, GIDD relies on a continuous interpolation-based pipeline with opaque interactions between uniform transitions and absorbing masks, which complicates hyperparameter tuning and hinders practical performance. In this work, we propose a Self-Correcting Discrete Diffusion (SCDD) model to reformulate pretrained self-correction with explicit state transitions and learn directly in discrete time. Our framework also simplifies the training noise schedule, eliminates a redundant remasking step, and relies exclusively on uniform transitions to learn self-correction. Experiments at the GPT-2 scale demonstrate that our method enables more efficient parallel decoding while preserving generation quality.