On the locality bias and results in the Long Range Arena

TL;DR

This paper analyzes the Long Range Arena benchmark, revealing that short-range dependencies largely drive performance and that architectural biases influence results, suggesting the need for a redesigned benchmark.

Contribution

It explains why architectures like SSMs outperform Transformers in LRA and demonstrates how training techniques can improve Transformer performance, challenging the benchmark's validity.

Findings

Most LRA performance stems from short-range dependencies.

Proper positional encoding enables Transformers to achieve state-of-the-art results.

Removing restrictions on SSM kernels does not decrease performance.

Abstract

The Long Range Arena (LRA) benchmark was designed to evaluate the performance of Transformer improvements and alternatives in long-range dependency modeling tasks. The Transformer and its main variants performed poorly on this benchmark, and a new series of architectures such as State Space Models (SSMs) gained some traction, greatly outperforming Transformers in the LRA. Recent work has shown that with a denoising pre-training phase, Transformers can achieve competitive results in the LRA with these new architectures. In this work, we discuss and explain the superiority of architectures such as MEGA and SSMs in the Long Range Arena, as well as the recent improvement in the results of Transformers, pointing to the positional and local nature of the tasks. We show that while the LRA is a benchmark for long-range dependency modeling, in reality most of the performance comes from short-range dependencies. Using training techniques to mitigate data inefficiency, Transformers are able to reach state-of-the-art performance with proper positional encoding. In addition, with the same techniques, we were able to remove all restrictions from SSM convolutional kernels and learn fully parameterized convolutions without decreasing performance, suggesting that the design choices behind SSMs simply added inductive biases and learning efficiency for these particular tasks. Our insights indicate that LRA results should be interpreted with caution and call for a redesign of the benchmark.