CORE: Context-Robust Remasking for Diffusion Language Models

TL;DR

CORE introduces a training-free inference-time revision method for diffusion language models that identifies and revises unstable tokens by probing their sensitivity to context perturbations, leading to more consistent generation.

Contribution

The paper presents CORE, a novel context-robust remasking framework that improves diffusion language model decoding without additional training by focusing on context-sensitive token revision.

Findings

Outperforms static confidence-based revision methods.

Achieves up to 9.2 percentage points improvement on MBPP.

Enhances reasoning and code benchmark performance.

Abstract

Standard decoding in Masked Diffusion Models (MDMs) is hindered by context rigidity: tokens are retained based on transient high confidence, often ignoring that early predictions lack full context. This creates cascade effects where initial inconsistencies misguide the remaining generation. Existing revision strategies attempt to mitigate this by relying on static confidence scores, but these signals are inherently myopic; inconsistent tokens can appear confident to the model itself. We propose Context-Robust Remasking (CORE), a training-free framework for inference-time revision. Rather than trusting static token probabilities, CORE identifies context-brittle tokens by probing their sensitivity to targeted masked-context perturbations. We formalize revision as a robust optimization objective over context shifts and efficiently approximate this objective to prioritize unstable tokens for revision. On LLaDA-8B-Base, CORE delivers consistent improvements across reasoning and code benchmarks, outperforming compute-matched baselines and improving MBPP by up to 9.2 percentage points.