Your Absorbing Discrete Diffusion Secretly Models the Bayesian Posterior

TL;DR

This paper reveals that discrete diffusion language models inherently approximate Bayesian posteriors, and introduces an ensemble inference method that provides calibrated uncertainty estimates without additional training.

Contribution

It proves the denoiser's output recovers the true Bayesian posterior and introduces a Monte Carlo ensemble method for improved inference in discrete diffusion models.

Findings

MC-marginal sampler closely matches analytic perplexity at K=128

Per-token variance strongly correlates with reconstruction error

Ensemble inference provides calibrated uncertainty estimates

Abstract

Discrete diffusion language models learn to reconstruct text from randomly masked inputs, yet under mild assumptions their denoiser already implements the exact Bayesian posterior over the original tokens. We prove that the expected denoiser output under the forward corruption distribution recovers the true posterior, and that a simple Monte Carlo estimator converges to this posterior at rate O(1/sqrt(K)) with finite-sample concentration bounds. Building on this insight, we introduce an inference-time ensemble that runs K independent denoising passes and aggregates both posterior means and variances without any extra training. On WikiText-2, our MC-marginal sampler recovers the analytic lambda-DCE zero-shot perplexity (approximately 39) to within a few points at K=128, and its per-token variance shows a strong rank correlation with reconstruction error (Spearman rho = 0.996). This cost-proportional procedure yields calibrated uncertainty estimates and a direct trade-off between compute and posterior fidelity in discrete diffusion LMs.