Plan, Verify and Fill: A Structured Parallel Decoding Approach for Diffusion Language Models

TL;DR

This paper introduces Plan-Verify-Fill (PVF), a novel decoding paradigm for diffusion language models that enhances efficiency by actively constructing semantic skeletons and verifying progress, significantly reducing computational effort while maintaining accuracy.

Contribution

PVF is a training-free, hierarchical decoding approach that leverages planning and validation to improve the efficiency of diffusion language models over existing methods.

Findings

Reduces NFE by up to 65% compared to confidence-based decoding.

Achieves superior efficiency without sacrificing accuracy.

Demonstrates effectiveness on LLaDA-8B-Instruct and Dream-7B-Instruct datasets.

Abstract

Diffusion Language Models (DLMs) present a promising non-sequential paradigm for text generation, distinct from standard autoregressive (AR) approaches. However, current decoding strategies often adopt a reactive stance, underutilizing the global bidirectional context to dictate global trajectories. To address this, we propose Plan-Verify-Fill (PVF), a training-free paradigm that grounds planning via quantitative validation. PVF actively constructs a hierarchical skeleton by prioritizing high-leverage semantic anchors and employs a verification protocol to operationalize pragmatic structural stopping where further deliberation yields diminishing returns. Extensive evaluations on LLaDA-8B-Instruct and Dream-7B-Instruct demonstrate that PVF reduces the Number of Function Evaluations (NFE) by up to 65% compared to confidence-based parallel decoding across benchmark datasets, unlocking superior efficiency without compromising accuracy.