Context-Aware Initialization for Reducing Generative Path Length in Diffusion Language Models

TL;DR

This paper introduces a context-aware initialization method for diffusion language models that reduces the number of denoising steps needed for inference by injecting prompt-conditioned priors, aiming to improve efficiency.

Contribution

It presents a training-free, prompt-conditioned prior injection approach for diffusion models, along with a confidence-based remasking mechanism to enhance initialization effectiveness.

Findings

Reduces denoising iterations by about 35% in experiments.

Naive warm-starting can sometimes decrease final accuracy.

Highlights the need for calibration and revision mechanisms.

Abstract

Diffusion Large Language Models (DLLMs) enable fully parallel token decoding but often remain impractical at inference time due to the many denoising iterations required to refine an information-free, fully masked initialization into coherent text. Most existing acceleration methods focus on traversing this generative trajectory more efficiently via improved solvers or sampling strategies. We advance a complementary perspective: shorten the trajectory itself by starting closer to the target distribution through context-aware initialization. We propose a training-free interface that injects prompt-conditioned priors from a lightweight auxiliary model into the diffusion initialization, and instantiate it with two mechanisms: discrete token injection and representation-level embedding interpolation. Because injected priors can be imperfect and unmask-only decoding can over-commit early, we also introduce a simple confidence-based remasking mechanism as a form of prior skepticism. Preliminary evidence on GSM8K suggests that context-aware initialization can substantially reduce denoising iterations (about 35\% fewer function evaluations in our setting), while also exposing a key open challenge: naive warm-starting can degrade final accuracy relative to strong diffusion baselines. We use these findings to motivate a research agenda around calibration, revision mechanisms, and representation alignment for reliable warm-started diffusion decoding.