Elastic Attention: Test-time Adaptive Sparsity Ratios for Efficient Transformers

TL;DR

Elastic Attention introduces a dynamic, input-dependent sparsity mechanism for transformers, significantly improving efficiency and performance in long-context scenarios by adapting attention modes during inference.

Contribution

We propose Elastic Attention, a novel method that dynamically adjusts attention sparsity ratios at test time using an Attention Router, enhancing scalability and adaptability of large language models.

Findings

Achieves strong performance with efficient inference on long-context benchmarks.

Enables dynamic adjustment of attention modes during inference.

Requires only 12 hours of training on 8xA800 GPUs.

Abstract

The quadratic complexity of standard attention mechanisms poses a significant scalability bottleneck for large language models (LLMs) in long-context scenarios. While hybrid attention strategies that combine sparse and full attention within a single model offer a viable solution, they typically employ static computation ratios (i.e., fixed proportions of sparse versus full attention) and fail to adapt to the varying sparsity sensitivities of downstream tasks during inference. To address this issue, we propose Elastic Attention, which allows the model to dynamically adjust its overall sparsity based on the input. This is achieved by integrating a lightweight Attention Router into the existing pretrained model, which dynamically assigns each attention head to different computation modes. Within only 12 hours of training on 8xA800 GPUs, our method enables models to achieve both strong performance and efficient inference. Experiments across three long-context benchmarks on widely-used LLMs demonstrate the superiority of our method.