Decoding Large Language Diffusion Models with Foreseeing Movement

TL;DR

This paper introduces the Foreseeing Decoding Method (FDM) for Large Language Diffusion Models, which optimizes decoding order by considering long-term impacts, improving efficiency and performance over heuristic approaches.

Contribution

The paper proposes FDM, a novel search-based decoding strategy that integrates local and global token considerations, and introduces FDM-A for faster, more efficient decoding.

Findings

FDM outperforms heuristic methods in diverse benchmarks.

FDM-A achieves better efficiency-performance balance.

Extensive experiments validate scalability and effectiveness.

Abstract

Large Language Diffusion Models (LLDMs) benefit from a flexible decoding mechanism that enables parallelized inference and controllable generations over autoregressive models. Yet such flexibility introduces a critical challenge: inference performance becomes highly sensitive to the decoding order of tokens. Existing heuristic methods, however, focus mainly on local effects while overlooking long-term impacts. To address this limitation, we propose the Foreseeing Decoding Method (FDM), a novel approach that integrates both local and global considerations to unlock the full potential, employing a search-based strategy to enable effective optimization in discrete spaces. Furthermore, by analyzing the consistency of chosen tokens in the full decoding process, we develop a variant, FDM with Acceleration (FDM-A), which restricts deep exploration to critical steps identified as the exploration and balance circumantences. Extensive experiments across diverse benchmarks and model architectures validate the scalability of FDM and demonstrate the superior efficiency-performance trade-off achieved by FDM-A. Our work might potentially provide a principled step toward more powerful decoding methods for LLDMs.