Early Decisions Matter: Proximity Bias and Initial Trajectory Shaping in Non-Autoregressive Diffusion Language Models

TL;DR

This paper analyzes the inference dynamics of non-autoregressive diffusion language models, revealing a proximity bias that influences initial token decisions and proposing a minimal intervention method to improve reasoning and planning tasks.

Contribution

The study identifies a proximity bias in non-autoregressive diffusion models and introduces a simple guiding approach to enhance their performance on complex tasks.

Findings

Proximity bias causes local error propagation in diffusion models.

Early token decisions critically influence the entire decoding trajectory.

The proposed method improves performance without added computational cost.

Abstract

Diffusion-based language models (dLLMs) have emerged as a promising alternative to autoregressive language models, offering the potential for parallel token generation and bidirectional context modeling. However, harnessing this flexibility for fully non-autoregressive decoding remains an open question, particularly for reasoning and planning tasks. In this work, we investigate non-autoregressive decoding in dLLMs by systematically analyzing its inference dynamics along the temporal axis. Specifically, we uncover an inherent failure mode in confidence-based non-autoregressive generation stemming from a strong proximity bias-the tendency for the denoising order to concentrate on spatially adjacent tokens. This local dependency leads to spatial error propagation, rendering the entire trajectory critically contingent on the initial unmasking position. Leveraging this insight, we present a minimal-intervention approach that guides early token selection, employing a lightweight planner and end-of-sequence temperature annealing. We thoroughly evaluate our method on various reasoning and planning tasks and observe substantial overall improvement over existing heuristic baselines without significant computational overhead.