Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning

TL;DR

This paper introduces the Ring-linear model series, a hybrid attention architecture that significantly reduces inference costs for long-context reasoning while maintaining state-of-the-art performance, and improves training efficiency with specialized hardware.

Contribution

The paper presents a novel hybrid attention architecture combining linear and softmax attention, optimized for long-context inference, with systematic exploration of attention ratios and high-performance training tools.

Findings

Reduces inference cost to 1/10 of dense models

Achieves over 50% reduction in inference cost compared to previous Ring models

Maintains state-of-the-art performance on complex reasoning benchmarks

Abstract

In this technical report, we present the Ring-linear model series, specifically including Ring-mini-linear-2.0 and Ring-flash-linear-2.0. Ring-mini-linear-2.0 comprises 16B parameters and 957M activations, while Ring-flash-linear-2.0 contains 104B parameters and 6.1B activations. Both models adopt a hybrid architecture that effectively integrates linear attention and softmax attention, significantly reducing I/O and computational overhead in long-context inference scenarios. Compared to a 32 billion parameter dense model, this series reduces inference cost to 1/10, and compared to the original Ring series, the cost is also reduced by over 50%. Furthermore, through systematic exploration of the ratio between different attention mechanisms in the hybrid architecture, we have identified the currently optimal model structure. Additionally, by leveraging our self-developed high-performance FP8 operator library-linghe, overall training efficiency has been improved by 50%. Benefiting from the high alignment between the training and inference engine operators, the models can undergo long-term, stable, and highly efficient optimization during the reinforcement learning phase, consistently maintaining SOTA performance across multiple challenging complex reasoning benchmarks.