Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling

TL;DR

DASH is a training-free method that improves long-context processing efficiency in LLMs and LMMs by halting redundant tokens based on their semantic stabilization, achieving speedups without accuracy loss.

Contribution

The paper introduces DASH, a novel token halting policy that leverages attention dynamics to reduce computational costs in long-context models without retraining.

Findings

DASH achieves significant prefill speedups across language and vision tasks.

DASH maintains model accuracy and hardware efficiency while halting redundant tokens.

The method generalizes well across different modalities and benchmarks.

Abstract

Prefilling computational costs pose a significant bottleneck for Large Language Models (LLMs) and Large Multimodal Models (LMMs) in long-context settings. While token pruning reduces sequence length, prior methods rely on heuristics that break compatibility with hardware-efficient kernels like FlashAttention. In this work, we observe that tokens evolve toward \textit{semantic fixing points}, making further processing redundant. To this end, we introduce Delta Attention Selective Halting (DASH), a training-free policy that monitors the layer-wise update dynamics of the self-attention mechanism to selectively halt stabilized tokens. Extensive evaluation confirms that DASH generalizes across language and vision benchmarks, delivering significant prefill speedups while preserving model accuracy and hardware efficiency. Code will be released at https://github.com/verach3n/DASH.git.