FASA: Frequency-aware Sparse Attention

TL;DR

FASA introduces a dynamic, query-aware token pruning method based on frequency-chunk analysis, significantly reducing memory use in large language models while maintaining high accuracy.

Contribution

FASA leverages a novel frequency-chunk sparsity insight to identify salient tokens, enabling efficient and robust attention pruning in long-context language modeling.

Findings

Achieves near full-KV performance with only 256 tokens on LongBench-V1.

Provides 2.56× speedup with 18.9% cache usage on AIME24.

Outperforms all token-eviction baselines across various long-context tasks.

Abstract

The deployment of Large Language Models (LLMs) faces a critical bottleneck when handling lengthy inputs: the prohibitive memory footprint of the Key Value (KV) cache. To address this bottleneck, the token pruning paradigm leverages attention sparsity to selectively retain a small, critical subset of tokens. However, existing approaches fall short, with static methods risking irreversible information loss and dynamic strategies employing heuristics that insufficiently capture the query-dependent nature of token importance. We propose FASA, a novel framework that achieves query-aware token eviction by dynamically predicting token importance. FASA stems from a novel insight into RoPE: the discovery of functional sparsity at the frequency-chunk (FC) level. Our key finding is that a small, identifiable subset of "dominant" FCs consistently exhibits high contextual agreement with the full attention head. This provides a robust and computationally free proxy for identifying salient tokens. Building on this insight, FASA first identifies a critical set of tokens using dominant FCs, and then performs focused attention computation solely on this pruned subset. Across a spectrum of long-context tasks, from sequence modeling to complex CoT reasoning, FASA consistently outperforms all token-eviction baselines and achieves near-oracle accuracy, demonstrating remarkable robustness even under constraint budgets. Notably, on LongBench-V1, FASA reaches nearly 100\% of full-KV performance when only keeping 256 tokens, and achieves 2.56$\times$ speedup using just 18.9\% of the cache on AIME24.