RaaS: Reasoning-Aware Attention Sparsity for Efficient LLM Reasoning

TL;DR

This paper introduces RaaS, a novel attention sparsity algorithm for large language models that efficiently identifies and retains key milestone tokens during reasoning tasks, balancing accuracy, time, and memory.

Contribution

RaaS leverages a new attention pattern to identify milestone tokens, enabling efficient reasoning with reduced complexity while maintaining high accuracy.

Findings

Achieves high accuracy with O(L) time and memory complexities.

Effectively identifies and retains milestone tokens during reasoning.

Balances the accuracy-time-memory trade-off in LLM reasoning tasks.

Abstract

Large Language Models (LLMs) have demonstrated strong capabilities across various domains, with recent advancements in challenging reasoning tasks such as mathematics and programming. However, solving reasoning tasks often requires an LLM to generate long sequences, incurring $O(N)$ time and memory complexities per token, where $N$ is the current sequence length. To reduce complexities, existing sparsity-based algorithms propose to retain Key-Value (KV) vectors, the intermediate representations of only the most critical tokens. However, these algorithms struggle with the "impossible trinity" of accuracy, time, and memory. For example, the state-of-the-art algorithm, Quest, achieves high accuracy with $O(L)$ time but $O(N)$ memory ($L$ is the cache budget, $L \ll N$). To address the "impossible trinity", in this paper, we identify a new attention pattern during the decode stage of reasoning tasks, where milestone tokens (analogous to lemmas in mathematical proofs) emerge, are utilized, and then become unimportant afterward. Based on this pattern, we propose a new algorithm RaaS that identifies milestone tokens and retains their KV vectors until they are no longer needed, achieving high accuracy with $O(L)$ time and $O(L)$ memory complexities.