Efficient Sparse Attention needs Adaptive Token Release

TL;DR

This paper introduces an adaptive token release mechanism for sparse attention in large language models, significantly improving computational efficiency while maintaining performance.

Contribution

It proposes a lightweight controller to adaptively release and rebuild key-value states, enabling efficient sparse attention in LLMs.

Findings

Achieves up to 221.8% throughput improvement.

Maintains competitive performance with full attention.

Demonstrates effectiveness in natural language generation.

Abstract

In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide array of text-centric tasks. However, their `large' scale introduces significant computational and storage challenges, particularly in managing the key-value states of the transformer, which limits their wider applicability. Therefore, we propose to adaptively release resources from caches and rebuild the necessary key-value states. Particularly, we accomplish this by a lightweight controller module to approximate an ideal top-$K$ sparse attention. This module retains the tokens with the highest top-$K$ attention weights and simultaneously rebuilds the discarded but necessary tokens, which may become essential for future decoding. Comprehensive experiments in natural language generation and modeling reveal that our method is not only competitive with full attention in terms of performance but also achieves a significant throughput improvement of up to 221.8%. The code for replication is available on the https://github.com/WHUIR/ADORE.