# Adaptive Attention Span in Transformers

## TL;DR

This paper introduces an adaptive attention span mechanism for Transformers that learns optimal context lengths, enabling larger context processing while controlling memory and computation, leading to state-of-the-art results in character-level language modeling.

## Contribution

It presents a new self-attention method that dynamically adjusts attention span, significantly extending context size in Transformers with efficient resource use.

## Key findings

- Achieved state-of-the-art performance on text8 and enwiki8 datasets.
- Extended maximum context size to 8,000 characters.
- Demonstrated efficient memory and computational control.

## Abstract

We propose a novel self-attention mechanism that can learn its optimal attention span. This allows us to extend significantly the maximum context size used in Transformer, while maintaining control over their memory footprint and computational time. We show the effectiveness of our approach on the task of character level language modeling, where we achieve state-of-the-art performances on text8 and enwiki8 by using a maximum context of 8k characters.