Beyond Confidence: Adaptive and Coherent Decoding for Diffusion Language Models

TL;DR

This paper introduces Coherent Contextual Decoding (CCD), an innovative inference framework for Diffusion Language Models that improves generation quality and speed by leveraging historical context and adaptive sampling strategies.

Contribution

The paper proposes a novel inference method for DLMs that enhances coherence and efficiency through trajectory rectification and adaptive unmasking based on a theoretical consistency metric.

Findings

Achieves up to 3.48x speedup in inference

Improves generation performance by 3.91%

Enhances trajectory coherence and sampling efficiency

Abstract

Diffusion Language Models (DLMs) have recently achieved significant success due to their any-order generation capabilities. However, existing inference methods typically rely on local, immediate-step metrics such as confidence or entropy which inherently lack a more reliable perspective. This limitation frequently leads to inconsistent sampling trajectories and suboptimal generation quality. To address this, we propose Coherent Contextual Decoding (CCD), a novel inference framework built upon two core innovations. First, CCD employs a trajectory rectification mechanism that leverages historical context to enhance sequence coherence, enabling the early rejection of suboptimal paths. We demonstrate that this mechanism is theoretically equivalent to modeling the consistency of historical steps via the conditional mutual information between context and token predictions. Building on this theoretical insight, we further address the inefficiency of conventional uniform decoding budgets. Instead of rigid allocations based on diffusion steps, we introduce an adaptive sampling strategy that dynamically adjusts the unmasking budget for each step according to our consistency metric. Consequently, our method significantly improves the quality of generation trajectories while accelerating the sampling process. Empirically, our method achieves a simultaneous enhancement in both inference speed and performance across diverse benchmarks on Dream and LLaDA, delivering up to 3.48x speedup alongside 3.91% performance improvement.