BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding

TL;DR

BLASST introduces a dynamic sparse attention mechanism for LLMs that accelerates inference by skipping negligible attention blocks using a simple threshold, achieving significant speedups without retraining.

Contribution

It provides a practical, hardware-friendly, and easy-to-integrate sparse attention method that improves inference speed in LLMs without additional training or pre-computation.

Findings

Achieves up to 1.52x speedup in prefill at 71.9% sparsity.

Achieves up to 1.48x speedup in decode at 73.2% sparsity.

Maintains benchmark accuracy while significantly reducing computation.

Abstract

The growing demand for long-context inference capabilities in Large Language Models (LLMs) has intensified the computational and memory bottlenecks inherent to the self-attention mechanism. To address this challenge, we introduce BLASST, a drop-in, dynamic sparse attention mechanism that accelerates inference by using only a fixed scalar threshold to skip attention blocks. Our method targets practical inference deployment by removing the barriers to adoption present in existing works. As such, BLASST eliminates training requirements, avoids expensive pre-computation passes, accelerates both prefill and decode across all major attention variants (MHA, GQA, MQA, and MLA), provides optimized support for modern hardware, and easily integrates into existing frameworks. This is achieved by reusing online softmax statistics to identify negligible attention scores, skipping softmax, value block loads, and the subsequent matrix multiplication. We demonstrate the BLASST algorithm by delivering optimized kernels with negligible latency overhead. Our automated threshold calibration procedure reveals a simple inverse relationship between optimal threshold and context length, meaning we require only a single threshold each for prefill and decode per model. Preserving benchmark accuracy, we demonstrate a 1.52x speedup for prefill at 71.9% sparsity and a 1.48x speedup for decode at 73.2% sparsity on modern GPUs.