Fast-FNet: Accelerating Transformer Encoder Models via Efficient Fourier Layers

TL;DR

Fast-FNet introduces efficient Fourier Transform-based methods to replace the attention mechanism in transformer encoders, reducing computational costs and enhancing performance for NLP tasks.

Contribution

The paper proposes novel Fourier Transform deployment strategies in transformer encoders, achieving smaller models, faster training, and improved efficiency over prior Fourier-based models.

Findings

Reduced training time and memory usage

Smaller model parameters

Performance improvements on benchmarks

Abstract

Transformer-based language models utilize the attention mechanism for substantial performance improvements in almost all natural language processing (NLP) tasks. Similar attention structures are also extensively studied in several other areas. Although the attention mechanism enhances the model performances significantly, its quadratic complexity prevents efficient processing of long sequences. Recent works focused on eliminating the disadvantages of computational inefficiency and showed that transformer-based models can still reach competitive results without the attention layer. A pioneering study proposed the FNet, which replaces the attention layer with the Fourier Transform (FT) in the transformer encoder architecture. FNet achieves competitive performances concerning the original transformer encoder model while accelerating training process by removing the computational burden of the attention mechanism. However, the FNet model ignores essential properties of the FT from the classical signal processing that can be leveraged to increase model efficiency further. We propose different methods to deploy FT efficiently in transformer encoder models. Our proposed architectures have smaller number of model parameters, shorter training times, less memory usage, and some additional performance improvements. We demonstrate these improvements through extensive experiments on common benchmarks.