What Context Features Can Transformer Language Models Use?

TL;DR

This paper investigates which aspects of long-range context are crucial for transformer language models' performance, finding that detailed syntactic content is less important than the overall context length.

Contribution

The study systematically measures the impact of different context features on transformer models, revealing that long context length is more vital than detailed syntactic information.

Findings

Long-range contexts are more important than detailed syntactic content.

Shuffling words within sentences minimally affects model performance.

Removing all but nouns from context reduces usable information by less than 15%.

Abstract

Transformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens. What aspects of these contexts contribute to accurate model prediction? We describe a series of experiments that measure usable information by selectively ablating lexical and structural information in transformer language models trained on English Wikipedia. In both mid- and long-range contexts, we find that several extremely destructive context manipulations -- including shuffling word order within sentences and deleting all words other than nouns -- remove less than 15% of the usable information. Our results suggest that long contexts, but not their detailed syntactic and propositional content, are important for the low perplexity of current transformer language models.