FNetAR: Mixing Tokens with Autoregressive Fourier Transforms

TL;DR

FNetAR introduces an autoregressive model replacing self-attention with Fourier transforms, achieving state-of-the-art language modeling performance with fewer layers, suggesting potential for parameter reduction in Transformer models.

Contribution

It presents a novel autoregressive approach using Fourier transforms instead of self-attention, demonstrating competitive performance with fewer layers.

Findings

FNetAR achieves 25.8 perplexity on Wikitext-103.

It outperforms Transformer-XL baseline with 24.2 perplexity.

The method suggests potential for reducing parameters in Transformer models.

Abstract

In this note we examine the autoregressive generalization of the FNet algorithm, in which self-attention layers from the standard Transformer architecture are substituted with a trivial sparse-uniformsampling procedure based on Fourier transforms. Using the Wikitext-103 benchmark, we demonstratethat FNetAR retains state-of-the-art performance (25.8 ppl) on the task of causal language modelingcompared to a Transformer-XL baseline (24.2 ppl) with only half the number self-attention layers,thus providing further evidence for the superfluity of deep neural networks with heavily compoundedattention mechanisms. The autoregressive Fourier transform could likely be used for parameterreduction on most Transformer-based time-series prediction models.