Don't Pay Attention

TL;DR

Avey is a novel neural architecture that efficiently processes arbitrarily long sequences by selecting relevant tokens, outperforming Transformers especially on long-range dependency tasks, while maintaining competitive short-range performance.

Contribution

Avey introduces a new architecture that decouples sequence length from context, enabling efficient long-range processing without attention or recurrence.

Findings

Avey outperforms Transformers on long-range dependency tasks.

Avey matches Transformer performance on short-range NLP benchmarks.

Avey processes arbitrarily long sequences efficiently.

Abstract

The Transformer has become the de facto standard for modern language models owing to its parallelizable training and effective autoregressive decoding. However, its fixed context window and the quadratic time and memory costs of its self-attention mechanism remain central bottlenecks. These constraints have revived interest in recurrent architectures that scale linearly with sequence length, but at the cost of reduced parallelism. In this paper, we introduce Avey, a new foundational architecture that breaks away from both attention and recurrence. Avey pairs a ranker with an autoregressive neural processor to select and contextualize only the most relevant tokens for any given token. Specifically, it decouples sequence length from context width, thus enabling effective and efficient processing of arbitrarily long sequences. Results show that Avey compares favorably to the Transformer across a variety of standard short-range NLP benchmarks, while significantly outperforming it on tasks requiring long-range dependency modeling.