Prompt Compression in Diffusion Large Language Models: Evaluating LLMLingua-2 on LLaDA

TL;DR

This paper evaluates how prompt compression affects diffusion large language models like LLaDA, revealing that compression impacts reasoning and summarization differently and highlighting the need for diffusion-specific compression methods.

Contribution

It provides the first systematic evaluation of prompt compression transferability from autoregressive to diffusion LLMs, demonstrating the limitations and challenges involved.

Findings

Summarization tasks are relatively robust under prompt compression.

Mathematical reasoning performance degrades significantly despite high semantic similarity.

Compression failures are mainly due to information omission rather than semantic drift.

Abstract

Prompt compression reduces inference cost and context length in large language models, but prior evaluations focus primarily on autoregressive architectures. This study investigates whether prompt compression transfers effectively to diffusion large language models (DLLMs) using LLMLingua-2, specifically the 8B-parameter DLLM LLaDA. We evaluate compression performance on GSM8K, DUC2004, and ShareGPT using 250 prompts per dataset at an approximate 2$\times$ compression ratio, across mathematical reasoning, prompt reconstruction, and summarization tasks. Outputs generated from original prompts, compressed prompts, reconstructed prompts, and reconstructed-prompt reasoning were compared using exact-match accuracy, BLEU, ROUGE, and BERTScore. Results show that semantic preservation does not necessarily imply stable downstream behavior in diffusion models. Summarization tasks remained comparatively robust under compression, while mathematical reasoning degraded substantially despite high semantic similarity scores. Reconstruction experiments further showed that semantically similar prompts may still omit reasoning-critical information required for stable denoising. Across tasks, BERTScore recall was consistently lower than precision, suggesting that compression failures are primarily driven by information omission rather than semantic drift. These findings indicate that prompt compression methods designed for autoregressive models do not transfer uniformly to diffusion large language models and motivate the development of diffusion-aware compression strategies.