Long Range Arena: A Benchmark for Efficient Transformers
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri,, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, Donald Metzler

TL;DR
This paper introduces the Long Range Arena benchmark to systematically evaluate efficient Transformer models on long-sequence tasks across multiple data types, addressing inconsistencies in current benchmarking practices.
Contribution
It proposes a unified benchmark suite for long-range Transformers, evaluates ten models systematically, and promotes standardized assessment in this research area.
Findings
Efficient Transformers show varied performance across tasks.
Benchmark reveals strengths and weaknesses of different models.
Provides a platform for future research and model improvement.
Abstract
Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this problem, more often than not claiming superior or comparable model quality to vanilla Transformer models. To this date, there is no well-established consensus on how to evaluate this class of models. Moreover, inconsistent benchmarking on a wide spectrum of tasks and datasets makes it difficult to assess relative model quality amongst many models. This paper proposes a systematic and unified benchmark, LRA, specifically focused on evaluating model quality under long-context scenarios. Our benchmark is a suite of tasks consisting of sequences ranging from to tokens, encompassing a wide range of data types and modalities such as text, natural, synthetic images,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsAdvanced Neural Network Applications · Ferroelectric and Negative Capacitance Devices · Power Line Inspection Robots
MethodsLinear Layer · Absolute Position Encodings · Position-Wise Feed-Forward Layer · Layer Normalization · Adam · Attention Is All You Need · Byte Pair Encoding · Dropout · Softmax · Multi-Head Attention
