Self-Indexing KVCache: Predicting Sparse Attention from Compressed Keys

TL;DR

This paper introduces a unified, hardware-efficient method for compressing and indexing key-value caches in self-attention, significantly improving sparse attention prediction for large language models.

Contribution

It proposes a sign-based 1-bit vector quantization scheme that combines compression and self-indexing, eliminating the need for external indices or complex predictors.

Findings

Reduces memory usage in self-attention caches

Integrates seamlessly with FlashAttention with minimal overhead

Achieves efficient sparse attention prediction in LLMs

Abstract

The KV cache in self-attention has emerged as a major bottleneck in long-context and large-batch inference for LLMs. Existing approaches often treat sparsity prediction and compression as separate modules, relying on auxiliary index structures to select relevant tokens, and on complex quantization schemes to reduce memory usage. This fragmented design introduces redundant overhead and limits scalability. In this paper, we propose a novel paradigm: treating the compressed key representation not merely as storage, but as a self-indexing structure that directly enables efficient sparse attention. By designing a sign-based 1-bit vector quantization (VQ) scheme, our method unifies compression and retrieval in a single, hardware-friendly format. This approach eliminates the need for external indices or learning-based predictors, offering a lightweight yet robust solution for memory-constrained inference. All components are designed to be hardware-efficient and easy to implement. By implementing custom CUDA kernels, our method integrates seamlessly with FlashAttention, minimizing additional runtime and memory overhead. Experimental results demonstrate that our approach delivers both effectiveness and efficiency.