Revisiting Linformer with a modified self-attention with linear complexity

TL;DR

This paper proposes a new self-attention method with linear complexity that is independent of hyperparameters, improving efficiency for long sequences in NLP, images, and audio tasks.

Contribution

A novel self-attention approach with linear time and space complexity that does not depend on the projection mapping dimension, enhancing scalability.

Findings

Reduces computational cost for long sequences

Applicable to images and audio data

Maintains performance without hyperparameter tuning

Abstract

Although Transformer models such as Google's BERT and OpenAI's GPT-3 are successful in many natural language processing tasks, training and deploying these models are costly and inefficient.Even if pre-trained models are used, deploying these models still remained a challenge due to their large size. Apart from deployment, these models take higher time during inference restricting user-friendliness. The main bottleneck is self-attention which uses quadratic time and space with respect to the sequence length. In order to reduce the quadratic time complexity of the self-attention mechanism, Linformer by Facebook's AI research team was introduced where they showed that the self-attention mechanism can be approximated by a low-rank matrix and exploiting this finding, a new method for self-attention with linear time and space complexity was proposed by them. In the Linformer, the time complexity depends on the projection mapping dimension which acts as a hyperparameter and affects the performance of the model, tuning this hyperparameter can be time-consuming. In this paper, I proposed an alternative method for self-attention with linear complexity in time and space and is independent of the projection mapping dimension. Since this method works for long sequences this can be used for images as well as audios.