A Systematic Analysis of Hybrid Linear Attention

TL;DR

This paper systematically evaluates various linear attention mechanisms and hybrid architectures in Transformers, revealing how different configurations impact language modeling and recall performance, and providing open-source models for future research.

Contribution

It offers a comprehensive analysis of linear and hybrid attention models, training and releasing 72 models across multiple variants and ratios for benchmarking and further study.

Findings

Superior standalone linear models do not always perform best in hybrids.

Recall improves significantly with more full attention layers, especially below a 3:1 ratio.

Gating mechanisms and hierarchical recurrence are key for effective hybrid models.

Abstract

Transformers face quadratic complexity and memory issues with long sequences, prompting the adoption of linear attention mechanisms using fixed-size hidden states. However, linear models often suffer from limited recall performance, leading to hybrid architectures that combine linear and full attention layers. Despite extensive hybrid architecture research, the choice of linear attention component has not been deeply explored. We systematically evaluate various linear attention models across generations - vector recurrences to advanced gating mechanisms - both standalone and hybridized. To enable this comprehensive analysis, we trained and open-sourced 72 models: 36 at 340M parameters (20B tokens) and 36 at 1.3B parameters (100B tokens), covering six linear attention variants across five hybridization ratios. Benchmarking on standard language modeling and recall tasks reveals that superior standalone linear models do not necessarily excel in hybrids. While language modeling remains stable across linear-to-full attention ratios, recall significantly improves with increased full attention layers, particularly below a 3:1 ratio. Our study highlights selective gating, hierarchical recurrence, and controlled forgetting as critical for effective hybrid models. We recommend architectures such as HGRN-2 or GatedDeltaNet with a linear-to-full ratio between 3:1 and 6:1 to achieve Transformer-level recall efficiently. Our models are open-sourced at https://huggingface.co/collections/m-a-p/hybrid-linear-attention-research-686c488a63d609d2f20e2b1e.