Linearizing Large Language Models

TL;DR

This paper introduces SUPRA, a cost-effective method to convert large pre-trained transformers into linear RNNs, enabling efficient training with minimal compute while maintaining competitive performance on benchmarks.

Contribution

The paper proposes SUPRA, a novel approach to linearize large pre-trained transformers into RNNs, reducing training costs by 95% and leveraging existing models' strengths.

Findings

Linear models perform competitively on benchmarks.

Persistent issues in in-context learning and long-context modeling.

SUPRA enables efficient linearization of large transformers.

Abstract

Linear transformers have emerged as a subquadratic-time alternative to softmax attention and have garnered significant interest due to their fixed-size recurrent state that lowers inference cost. However, their original formulation suffers from poor scaling and underperforms compute-matched transformers. Recent linear models such as RWKV and Mamba have attempted to address these shortcomings by proposing novel time-mixing and gating architectures, but pre-training large language models requires significant data and compute investments. Thus, the search for subquadratic architectures is limited by the availability of compute and quality pre-training datasets. As a cost-effective alternative to pre-training linear transformers, we propose Scalable UPtraining for Recurrent Attention (SUPRA). We present a method to uptrain existing large pre-trained transformers into Recurrent Neural Networks (RNNs) with a modest compute budget. This allows us to leverage the strong pre-training data and performance of existing transformer LLMs, while requiring 5% of the training cost. We find that our linearization technique leads to competitive performance on standard benchmarks, but we identify persistent in-context learning and long-context modeling shortfalls for even the largest linear models. Our code and models can be found at https://github.com/TRI-ML/linear_open_lm.