Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models

TL;DR

This paper introduces NAtS-L, a hybrid attention framework that adaptively combines linear and softmax attention at the token level, improving efficiency and expressivity for long-context models.

Contribution

The paper proposes a novel token-level hybrid attention model that dynamically selects between linear and softmax attention, enhancing efficiency and expressivity in long-context transformers.

Findings

NAtS-L achieves better efficiency compared to pure softmax models.

The hybrid approach maintains high expressivity for long-term dependencies.

Experimental results demonstrate improved performance on long-context tasks.

Abstract

The quadratic computational complexity of softmax transformers has become a bottleneck in long-context scenarios. In contrast, linear attention model families provide a promising direction towards a more efficient sequential model. These linear attention models compress past KV values into a single hidden state, thereby efficiently reducing complexity during both training and inference. However, their expressivity remains limited by the size of their hidden state. Previous work proposed interleaving softmax and linear attention layers to reduce computational complexity while preserving expressivity. Nevertheless, the efficiency of these models remains bottlenecked by their softmax attention layers. In this paper, we propose Neural Attention Search Linear (NAtS-L), a framework that applies both linear attention and softmax attention operations within the same layer on different tokens. NAtS-L automatically determines whether a token can be handled by a linear attention model, i.e., tokens that have only short-term impact and can be encoded into fixed-size hidden states, or require softmax attention, i.e., tokens that contain information related to long-term retrieval and need to be preserved for future queries. By searching for optimal Gated DeltaNet and softmax attention combinations across tokens, we show that NAtS-L provides a strong yet efficient token-level hybrid architecture.