Longformer: The Long-Document Transformer
Iz Beltagy, Matthew E. Peters, Arman Cohan

TL;DR
The paper introduces Longformer, a transformer model with a linear-scalable attention mechanism, enabling efficient processing of long documents and achieving state-of-the-art results on multiple long-sequence tasks.
Contribution
It proposes a novel attention mechanism combining local and global attention, and introduces Longformer-Encoder-Decoder for long document generation tasks.
Findings
Achieves state-of-the-art on text8 and enwik8 character-level language modeling.
Outperforms RoBERTa on long document tasks after pretraining and fine-tuning.
Demonstrates effectiveness of Longformer-Encoder-Decoder on arXiv summarization.
Abstract
Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer's attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
- 🤗allenai/longformer-base-4096model· 1.2M dl· ♡ 2251.2M dl♡ 225
- 🤗bigcode/starcoder2-15bmodel· 5.2k dl· ♡ 6655.2k dl♡ 665
- 🤗allenai/led-base-16384model· 23k dl· ♡ 5023k dl♡ 50
- 🤗allenai/led-large-16384-arxivmodel· 742 dl· ♡ 34742 dl♡ 34
- 🤗allenai/led-large-16384model· 16k dl· ♡ 3116k dl♡ 31
- 🤗allenai/longformer-base-4096-extra.pos.embd.onlymodel· 3 dl· ♡ 13 dl♡ 1
- 🤗jpwahle/longformer-base-plagiarism-detectionmodel· 333 dl· ♡ 17333 dl♡ 17
- 🤗mrm8488/longformer-base-4096-finetuned-squadv2model· 481 dl· ♡ 15481 dl♡ 15
- 🤗mrm8488/longformer-base-4096-spanishmodel· 466 dl· ♡ 16466 dl♡ 16
- 🤗valhalla/longformer-base-4096-finetuned-squadv1model· 212 dl· ♡ 22212 dl♡ 22
Videos
How much memory does Longformer use?· youtube
Longformer: The Long-Document Transformer· youtube
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Computational Physics and Python Applications
MethodsLinear Layer · Multi-Head Attention · Residual Connection · Attention Dropout · WordPiece · Linear Warmup With Linear Decay · Refunds@Expedia|||How do I get a full refund from Expedia? · Weight Decay · Dense Connections · AdamW
