Improving the Throughput of Diffusion-based Large Language Models via a Training-Free Confidence-Aware Calibration

TL;DR

CadLLM is a training-free, confidence-aware method that dynamically adjusts generation parameters to significantly improve inference throughput of diffusion-based large language models without sacrificing accuracy.

Contribution

It introduces a novel, lightweight adaptive approach for diffusion-based LLMs that controls generation parameters based on confidence, enhancing throughput without retraining.

Findings

Achieves up to 2.28x throughput improvement over state-of-the-art methods.

Demonstrates effectiveness across four popular tasks.

Maintains competitive accuracy with increased efficiency.

Abstract

We present CadLLM, a training-free method to accelerate the inference throughput of diffusion-based LLMs (dLLMs). We first investigate the dynamic nature of token unmasking confidence across blocks and steps. Based on this observation, we present a lightweight adaptive approach that controls the generation block size, step size, and threshold based on the average confidence of unmasked tokens. We further reduce softmax overhead by dynamically leveraging a subset of the vocabulary to regulate sampling breadth. CadLLM is a plug-and-play, model-agnostic method compatible with KV-cache-based dLLMs. Extensive experiments on four popular tasks demonstrate that CadLLM yields up to 1.1-2.28x throughput improvement over the state-of-the-art baseline with competitive accuracy.