Divide and Conquer: Accelerating Diffusion-Based Large Language Models via Adaptive Parallel Decoding

TL;DR

This paper introduces DiCo, an adaptive parallel decoding method for diffusion-based large language models that significantly accelerates inference by exploiting inherent parallelism while preserving generation quality.

Contribution

The paper proposes a novel divide-and-conquer decoding paradigm, DiCo, that bridges the gap between theoretical parallelism and practical performance in diffusion-based LLMs.

Findings

DiCo achieves substantial inference speedups.

DiCo maintains competitive generation quality.

The method effectively explores and utilizes model parallelism.

Abstract

Diffusion-based large language models (dLLMs) have shown promising performance across various reasoning tasks, establishing themselves as an alternative to autoregressive large language models (LLMs). Unlike autoregressive LLMs that generate one token per step based on all previous tokens, dLLMs theoretically enable parallel generation of multiple tokens at each decoding step. However, recent dLLMs still favor one-token-per-step generation in practice, as directly decoding multiple masked tokens often leads to degraded generation quality and stability. This reveals a substantial gap between the theoretical parallelism and practical performance of dLLMs. To bridge this gap, we introduce an adaptive parallel decoding approach, namely DiCo, which features a three-phase divide-and-conquer paradigm to unleash the inherent parallelism of dLLMs. During the Divide phase, DiCo first explores the input masked sequence and identifies masked tokens as seed tokens, which are then expanded to construct a set of local clusters. During the Conquer phase, DiCo performs parallel decoding across different local clusters constructed in the Divide phase. The divide-and-conquer process repeatedly alternates between the Divide and Conquer phases until convergence. During the Finalize phase, DiCo decodes the remaining few masked tokens using an effective fine-grained compound decoding scheme to finalize the generation. Extensive experiments demonstrate that DiCo can achieve significant inference speedups while maintaining competitive generation quality.