Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling

TL;DR

This paper reveals that masked diffusion models are essentially time-agnostic masked models, introduces a faster sampling method, and questions their superiority over auto-regressive models due to numerical issues affecting sampling accuracy.

Contribution

The paper uncovers the time-agnostic nature of MDMs, proposes a faster first-hitting sampler, and identifies numerical sampling issues impacting their performance evaluation.

Findings

FHS achieves 20× speedup over traditional sampling.

MDMs are theoretically equivalent to masked models, not diffusion models.

Numerical issues lower token diversity and affect evaluation metrics.

Abstract

Masked diffusion models (MDMs) have emerged as a popular research topic for generative modeling of discrete data, thanks to their superior performance over other discrete diffusion models, and are rivaling the auto-regressive models (ARMs) for language modeling tasks. The recent effort in simplifying the masked diffusion framework further leads to alignment with continuous-space diffusion models and more principled training and sampling recipes. In this paper, however, we reveal that both training and sampling of MDMs are theoretically free from the time variable, arguably the key signature of diffusion models, and are instead equivalent to masked models. The connection on the sampling aspect is drawn by our proposed first-hitting sampler (FHS). Specifically, we show that the FHS is theoretically equivalent to MDMs' original generation process while significantly alleviating the time-consuming categorical sampling and achieving a 20$\times$ speedup. In addition, our investigation raises doubts about whether MDMs can truly beat ARMs in text generation. We identify, for the first time, an underlying numerical issue, even with the commonly used 32-bit floating-point precision, which results in inaccurate categorical sampling. We show that it lowers the effective temperature both theoretically and empirically, and the resulting decrease in token diversity makes previous evaluations, which assess the generation quality solely through the incomplete generative perplexity metric, somewhat unfair.