SpecAttn: Co-Designing Sparse Attention with Self-Speculative Decoding

TL;DR

SpecAttn introduces a verification-guided sparse attention mechanism for large language models, dynamically selecting critical KV entries during decoding, significantly boosting inference speed and efficiency.

Contribution

It presents a novel method that integrates verification into sparse attention, enabling dynamic critical KV entry selection during decoding for improved speed.

Findings

Achieves 2.81× higher throughput over standard decoding.

Outperforms existing sparsity-based self-speculative decoding methods by 1.29×.

Reduces KV selection overhead while maintaining high draft acceptance rates.

Abstract

Long-context large language model (LLM) inference has become the norm for today's AI applications. However, it is severely bottlenecked by the increasing memory demands of its KV cache. Previous works have shown that self-speculative decoding with sparse attention, where tokens are drafted using a subset of the KV cache and verified in parallel with full KV cache, speeds up inference in a lossless way. However, this approach relies on standalone KV selection algorithms to select the KV entries used for drafting and overlooks that the criticality of each KV entry is inherently computed during verification. In this paper, we propose SpecAttn, a self-speculative decoding method with verification-guided sparse attention. SpecAttn identifies critical KV entries as a byproduct of verification and only loads these entries when drafting subsequent tokens. This not only improves draft token acceptance rate but also incurs low KV selection overhead, thereby improving decoding throughput. SpecAttn achieves 2.81$\times$ higher throughput over vanilla auto-regressive decoding and 1.29$\times$ improvement over state-of-the-art sparsity-based self-speculative decoding methods.