Retrospective Sparse Attention for Efficient Long-Context Generation

TL;DR

RetroAttention introduces a retrospective KV cache update method for LLMs, improving long-context generation efficiency and accuracy by correcting past attention outputs with minimal latency overhead.

Contribution

It presents RetroAttention, a novel technique for revising past attention outputs using new KV entries, enhancing long-context generation in LLMs.

Findings

Outperforms state-of-the-art KV compression methods

Increases effective KV exposure by up to 1.6×

Improves accuracy by up to 21.9%

Abstract

Large Language Models (LLMs) are increasingly deployed in long-context tasks such as reasoning, code generation, and multi-turn dialogue. However, inference over extended contexts is bottlenecked by the Key-Value (KV) cache, whose memory footprint grows linearly with sequence length and dominates latency at each decoding step. While recent KV cache compression methods identify and load important few tokens, they focus predominantly on input contexts and fail to address the cumulative attention errors that arise during long decoding. In this paper, we introduce RetroAttention, a novel KV cache update technique that retrospectively revises past attention outputs using newly arrived KV entries from subsequent decoding steps. By maintaining a lightweight output cache, RetroAttention enables past queries to be efficiently supplemented with more contexts, while incurring minimal latency overhead. This breaks the fixed-attention-output paradigm and allows continual correction of prior approximations. Extensive experiments on long-generation benchmarks show that RetroAttention consistently outperforms state-of-the-art (SOTA) KV compression methods, increasing effective KV exposure by up to 1.6$\times$ and accuracy by up to 21.9\%.