Swordsman: Entropy-Driven Adaptive Block Partition for Efficient Diffusion Language Models

TL;DR

Swordsman introduces an entropy-driven adaptive block partitioning method for diffusion language models, improving inference efficiency and quality by aligning decoding with semantic boundaries without additional training.

Contribution

It proposes a novel entropy-based adaptive block partitioning framework that dynamically aligns decoding with semantic boundaries in diffusion language models.

Findings

Achieves state-of-the-art performance in efficiency and quality

Effectively aligns decoding with semantic or syntactic boundaries

Operates without additional training, supported by KV Cache

Abstract

Block-wise decoding effectively improves the inference speed and quality in diffusion language models (DLMs) by combining inter-block sequential denoising and intra-block parallel unmasking. However, existing block-wise decoding methods typically partition blocks in a rigid and fixed manner, which inevitably fragments complete semantic or syntactic constituents, leading to suboptimal performance. Inspired by the entropy reduction hypothesis (ERH), we recognize that constituent boundaries offer greater opportunities for uncertainty reduction, which motivates us to employ entropy analysis for identifying constituent boundaries. Therefore, we propose Swordsman, an entropy-driven adaptive block-wise decoding framework for DLMs. Swordsman adaptively partitions blocks by identifying entropy shifts between adjacent tokens to better align with semantic or syntactic constituent boundaries. In addition, Swordsman dynamically adjusts unmasking thresholds conditioned on the real-time unmasking status within a block, further improving both efficiency and stability. As a training-free framework, supported by KV Cache, Swordsman demonstrates state-of-the-art performance across extensive evaluations.