DARE: Diffusion Language Model Activation Reuse for Efficient Inference

TL;DR

DARE introduces token-wise activation reuse techniques for diffusion language models, significantly reducing inference latency while maintaining high output quality, and can be combined with existing methods for further efficiency gains.

Contribution

It proposes novel token-wise reuse mechanisms (DARE-KV and DARE-O) for diffusion LLMs, improving efficiency without retraining and with minimal performance loss.

Findings

Up to 1.20x per-layer latency reduction.

Reuses up to 87% of attention activations.

Negligible degradation on reasoning and code-generation benchmarks.

Abstract

Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to auto-regressive (AR) models, offering greater expressive capacity and potential for parallel generation and faster inference. However, open-source dLLMs remain immature, lagging behind AR models in both efficiency and quality. We identify an underexplored property of dLLMs: *token-wise redundancy* in bi-directional self-attention. Self-attention activations are highly correlated across tokens, and temporal changes in query representations can predict redundancy in corresponding key, value, and output activations. We introduce DARE, with two complementary mechanisms: DARE-KV, which reuses cached key-value (KV) activations, and DARE-O, which reuses output activations to reduce redundant computation while preserving quality. DARE achieves up to 1.20x per-layer latency reduction and reuses up to 87% of attention activations, with negligible degradation on reasoning and code-generation benchmarks. DARE-KV and DARE-O incur average performance drops of only 2.0% and 1.2%, respectively. Combined with techniques such as prefix caching and Fast-dLLM, DARE provides additive gains without retraining. These results establish token-wise reuse as an effective strategy for improving the efficiency of diffusion-based LLMs while preserving generation fidelity. Code: https://github.com/enyac-group/DARE