Focus on the Core: Empowering Diffusion Large Language Models by Self-Contrast

TL;DR

This paper introduces FoCore, a novel decoding strategy for Diffusion Large Language Models that leverages high-information-density tokens to improve output quality and decoding efficiency without additional training.

Contribution

The authors propose FoCore and FoCore extunderscore A, training-free methods that enhance generation quality and speed by focusing on high-information-density tokens during decoding.

Findings

FoCore improves performance on math, code, and reasoning benchmarks.

FoCore extunderscore A accelerates decoding by 2.07x and reduces latency by 58.4%.

Explicitly conditioning on HD tokens enhances output quality.

Abstract

The iterative denoising paradigm of Diffusion Large Language Models (DLMs) endows them with a distinct advantage in global context modeling. However, current decoding strategies fail to leverage this capability, typically exhibiting a local preference that overlooks the heterogeneous information density within the context, ultimately degrading generation quality. To address this limitation, we systematically investigate high-information-density (HD) tokens and present two key findings: (1) explicitly conditioning on HD tokens substantially improves output quality; and (2) HD tokens exhibit an early-decoding tendency, converging earlier than surrounding tokens. Motivated by these findings, we propose Focus on the Core \textbf{(FoCore)}, a training-free decoding strategy that utilizes HD tokens in a self-contrast manner, wherein HD tokens are temporarily remasked as negative samples, to guide generation. We further introduce FoCore\_Accelerate \textbf{(FoCore\_A)}, an efficient variant that, upon detecting HD token convergence, performs parallel decoding over stable candidates within a local context window, substantially accelerating generation. Extensive experiments on math, code and logical reasoning benchmarks demonstrate that FoCore consistently improves generation quality and efficiency across both LLaDA and Dream backbones. For instance, on HumanEval, FoCore improves pass@1 from 39.02 to 42.68 over standard Classifier-Free Guidance, while FoCore-A reduces the number of decoding steps by 2.07x and per-sample latency from 20.76s to 8.64s (-58.4\%).