Big Bird: Transformers for Longer Sequences

TL;DR

BigBird introduces a sparse attention mechanism for transformers that reduces memory complexity from quadratic to linear, enabling processing of much longer sequences while maintaining theoretical properties and improving NLP and genomics task performance.

Contribution

It proposes a novel sparse attention mechanism for transformers that allows handling longer sequences with linear complexity, preserving universality and Turing completeness.

Findings

Handles sequences up to 8x longer than previous models.

Improves performance on question answering and summarization tasks.

Enables new applications in genomics data analysis.

Abstract

Transformers-based models, such as BERT, have been one of the most successful deep learning models for NLP. Unfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence length due to their full attention mechanism. To remedy this, we propose, BigBird, a sparse attention mechanism that reduces this quadratic dependency to linear. We show that BigBird is a universal approximator of sequence functions and is Turing complete, thereby preserving these properties of the quadratic, full attention model. Along the way, our theoretical analysis reveals some of the benefits of having $O(1)$ global tokens (such as CLS), that attend to the entire sequence as part of the sparse attention mechanism. The proposed sparse attention can handle sequences of length up to 8x of what was previously possible using similar hardware. As a consequence of the capability to handle longer context, BigBird drastically improves performance on various NLP tasks such as question answering and summarization. We also propose novel applications to genomics data.