Latent Shadows: The Gaussian-Discrete Duality in Masked Diffusion

TL;DR

This paper establishes a theoretical duality between masked discrete diffusion and Gaussian processes, enabling a new deterministic distillation method that significantly speeds up inference in language models without quality loss.

Contribution

It introduces Masked Diffusion Duality and Masked Consistency Distillation, providing a novel theoretical framework and practical algorithm for efficient deterministic sampling in masked diffusion models.

Findings

Achieved 16× inference speedup over stochastic methods.

Established explicit duality between masked diffusion and Gaussian processes.

Demonstrated improved efficiency without sacrificing generation quality.

Abstract

Masked discrete diffusion is a dominant paradigm for high-quality language modeling where tokens are iteratively corrupted to a mask state, yet its inference efficiency is bottlenecked by the lack of deterministic sampling tools. While diffusion duality enables deterministic distillation for uniform models, these approaches generally underperform masked models and rely on complex integral operators. Conversely, in the masked domain, prior methods typically assume the absence of deterministic trajectories, forcing a reliance on stochastic distillation. To bridge this gap, we establish explicit Masked Diffusion Duality, proving that the masked process arises as the projection of a continuous Gaussian process via a novel maximum-value index preservation mechanism. Furthermore, we introduce Masked Consistency Distillation (MCD), a principled framework that leverages this duality to analytically construct the deterministic coupled trajectories required for consistency distillation, bypassing numerical ODE solvers. This result strictly improves upon prior stochastic distillation methods, achieving a 16$\times$ inference speedup without compromising generation quality. Our findings not only provide a solid theoretical foundation connecting masked and continuous diffusion, but also unlock the full potential of consistency distillation for high-performance discrete generation. Our code is available at https://anonymous.4open.science/r/MCD-70FD.