Mixing Times of Glauber Dynamics on Masked Language Models

TL;DR

This paper models the iterative token resampling in masked language models as a Glauber dynamics Markov chain, analyzing its mixing times and stationary behavior to understand the global distributional effects of local conditionals.

Contribution

It introduces a rectangle test to certify incompatibility of MLM conditionals, provides theoretical bounds on mixing times, and empirically demonstrates phase transitions and semantic basin structures.

Findings

MLM conditionals are often incompatible, verified by the rectangle test.

High-temperature regimes have rapid mixing with O(n log n) time.

Low-temperature regimes exhibit metastability with slow escape from semantic basins.

Abstract

Masked language models (MLMs) define local conditional distributions over tokens but do not, in general, correspond to any consistent joint distribution over sequences. This raises a fundamental question: what global distributional behavior is induced when such conditionals are used iteratively for generation? We address this question by modeling iterative masked-token resampling as a Glauber dynamics Markov chain on the discrete space of token sequences. We first show that MLM conditionals are intrinsically incompatible: we introduce a rectangle test that certifies this incompatibility and empirically verify its prevalence across modern MLMs. We then provide a theoretical analysis of the induced Markov chain. Under bounded cross-token influence, we establish a high-temperature contraction result implying $O(n\log n)$ mixing time where $n$ is the sequence length. In contrast, we prove that under a uniform local margin condition, the chain exhibits metastability, with exponentially slow escape from semantic basins at low temperatures. Empirically, we demonstrate a phase transition in mixing behavior as a function of temperature and sequence length, consistent with the theoretical predictions. We further characterize the induced stationary behavior through semantic trajectories, identifying persistent structures such as long-lived traps and recurrent semantic basins, with political content serving as a measurable case study.