Do Long-Range Language Models Actually Use Long-Range Context?

TL;DR

This paper investigates whether long-range Transformer language models effectively utilize extended context, revealing limited benefits beyond copying and domain-specific advantages, especially in literary texts.

Contribution

The study provides a detailed analysis of long-range Transformer models, showing their limited use of extended context and highlighting domain differences in long-range context utility.

Findings

Long-range context improves predictions mainly for copying tokens.

Models do not benefit from long context for sentence-level tasks.

Long-range context is most helpful for literary novels, less so for textbooks or magazines.

Abstract

Language models are generally trained on short, truncated input sequences, which limits their ability to use discourse-level information present in long-range context to improve their predictions. Recent efforts to improve the efficiency of self-attention have led to a proliferation of long-range Transformer language models, which can process much longer sequences than models of the past. However, the ways in which such models take advantage of the long-range context remain unclear. In this paper, we perform a fine-grained analysis of two long-range Transformer language models (including the \emph{Routing Transformer}, which achieves state-of-the-art perplexity on the PG-19 long-sequence LM benchmark dataset) that accept input sequences of up to 8K tokens. Our results reveal that providing long-range context (i.e., beyond the previous 2K tokens) to these models only improves their predictions on a small set of tokens (e.g., those that can be copied from the distant context) and does not help at all for sentence-level prediction tasks. Finally, we discover that PG-19 contains a variety of different document types and domains, and that long-range context helps most for literary novels (as opposed to textbooks or magazines).