Generation Order and Parallel Decoding in Masked Diffusion Models: An Information-Theoretic Perspective

TL;DR

This paper introduces an information-theoretic framework to analyze generation order and parallel decoding in Masked Diffusion Models, revealing fundamental trade-offs and failure modes that impact inference quality and efficiency.

Contribution

It provides a unified theoretical analysis of order sensitivity and parallelization bias in MDMs, highlighting intrinsic errors and costs associated with different decoding strategies.

Findings

Easy-First decoding benefits increase with model error.

Parallel decoding can cause large distributional divergence.

Verification reduces sampling error but with exponential cost.

Abstract

Masked Diffusion Models (MDMs) significantly accelerate inference by trading off sequential determinism. However, the theoretical mechanisms governing generation order and the risks inherent in parallelization remain under-explored. In this work, we provide a unified information-theoretic framework to decouple and analyze two fundamental sources of failure: order sensitivity and parallelization bias. Our analysis yields three key insights: (1) The benefits of Easy-First decoding (prioritizing low-entropy tokens) are magnified as model error increases; (2) factorized parallel decoding introduces intrinsic sampling errors that can lead to arbitrary large Reverse KL divergence, capturing "incoherence" failures that standard Forward KL metrics overlook; and (3) while verification can eliminate sampling error, it incurs an exponential cost governed by the total correlation within a block. Conversely, heuristics like remasking, though computationally efficient, cannot guarantee distributional correctness. Experiments on a controlled Block-HMM and large-scale MDMs (LLaDA) for arithmetic reasoning validate our theoretical framework.