Higher Order Linear Transformer

TL;DR

This paper extends the linear transformer model by introducing a second-order approximation of the softmax normalization, aiming to improve efficiency while maintaining performance.

Contribution

It presents a novel second-order approximation method for the attention mechanism in linear transformers, building upon previous linearization techniques.

Findings

Achieves linear complexity in attention computation

Maintains comparable performance to standard transformers

Demonstrates improved efficiency in large-scale models

Abstract

Following up on the linear transformer part of the article from Katharopoulos et al., that takes this idea from Shen et al., the trick that produces a linear complexity for the attention mechanism is re-used and extended to a second-order approximation of the softmax normalization.