CSAttention: Centroid-Scoring Attention for Accelerating LLM Inference

TL;DR

CSAttention is a novel sparse attention method designed for high-throughput, long-context LLM inference, significantly improving speed while maintaining accuracy by front-loading computation and using query-centric lookup tables.

Contribution

It introduces a training-free, offline prefill-based sparse attention technique optimized for long contexts and high sparsity, outperforming existing methods in speed and accuracy.

Findings

Achieves up to 4.6x inference speedup at 128K context length.

Maintains near-identical accuracy to full attention.

Outperforms state-of-the-art sparse attention methods in experiments.

Abstract

Long-context LLMs increasingly rely on extended, reusable prefill prompts for agents and domain Q&A, pushing attention and KV-cache to become the dominant decode-time bottlenecks. While sparse attention reduces computation and transfer costs, it often struggles to maintain accuracy at high sparsity levels due to the inherent distribution shift between Queries and Keys. We propose Centroid-Scoring Attention (CSAttention), a training-free sparse attention method optimized for high-throughput serving of reusable contexts. CSAttention adopts a storage-for-computation strategy tailored to the offline-prefill/online-decode setting: it front-loads computation into a one-time offline prefill phase that can be amortized across multiple queries, while aggressively optimizing per-step decoding latency. Specifically, CSAttention constructs query-centric lookup tables during offline prefill, whose size remains fixed during decoding, and enables online decoding to replace full-context scans with efficient table lookups and GPU-friendly score accumulation. Extensive experiments demonstrate that CSAttention achieves near-identical accuracy to full attention. Under high sparsity (95%) and long-context settings (32K-128K), CSAttention consistently outperforms state-of-the-art sparse attention methods in both model accuracy and inference speed, achieving up to 4.6x inference speedup over the most accurate baseline at a context length of 128K.