Compressive Transformers for Long-Range Sequence Modelling

TL;DR

The paper introduces the Compressive Transformer, a novel sequence model that compresses past memories to effectively handle long-range dependencies, achieving state-of-the-art results in language modeling and other tasks.

Contribution

It presents the first comprehensive approach to compressive memory in transformers, improving long-range sequence modeling across multiple domains.

Findings

Achieves 17.1 perplexity on WikiText-103

Achieves 0.97 bits per character on Enwik8

Effective in high-frequency speech modeling and reinforcement learning tasks

Abstract

We present the Compressive Transformer, an attentive sequence model which compresses past memories for long-range sequence learning. We find the Compressive Transformer obtains state-of-the-art language modelling results in the WikiText-103 and Enwik8 benchmarks, achieving 17.1 ppl and 0.97 bpc respectively. We also find it can model high-frequency speech effectively and can be used as a memory mechanism for RL, demonstrated on an object matching task. To promote the domain of long-range sequence learning, we propose a new open-vocabulary language modelling benchmark derived from books, PG-19.