SimpleTRON: Simple Transformer with O(N) Complexity

TL;DR

SimpleTRON introduces a linear-time Transformer variant that eliminates the need for quadratic attention, offering faster performance and better transferability from pretrained models, outperforming existing approximation methods on long-range tasks.

Contribution

It proposes a novel linear Transformer model that removes the quadratic attention complexity without approximation, enabling efficient long-range data processing.

Findings

Outperforms existing sub-quadratic attention models on Long-Range Arena benchmark.

Can transfer weights from pretrained large language models.

Operates with O(N) complexity, significantly reducing computational costs.

Abstract

In this paper, we propose that the dot product pairwise matching attention layer, which is widely used in Transformer-based models, is redundant for the model performance. Attention, in its original formulation, has to be seen rather as a human-level tool to explore and/or visualize relevancy scores in sequential data. However, the way how it is constructed leads to significant computational complexity. Instead, we present SimpleTRON: Simple Transformer with O(N) Complexity, a simple and fast alternative without any approximation that, unlike other approximation models, does not have any architecture-related overhead and therefore can be seen as a purely linear Transformer-like model. This architecture, to the best of our knowledge, outperforms existing sub-quadratic attention approximation models on several tasks from the Long-Range Arena benchmark. Moreover, we show, that SimpleTRON can benefit from weight transfer from pretrained large language models, as its parameters can be fully transferable.