What Kinds of Tokens Benefit from Distant Text? An Analysis on Long Context Language Modeling

TL;DR

This paper investigates which types of words benefit most from long contexts in language models, revealing that content words, initial tokens, and frequent patterns are most influenced, with models becoming more confident as context length increases.

Contribution

The study provides a detailed analysis of how different token types and prior knowledge affect language model predictions with increasing context length.

Findings

Content words and initial tokens benefit most from long contexts.

Frequent N-grams significantly influence token predictions.

Longer contexts increase model confidence and probability sharpness.

Abstract

As the context length that large language models can handle continues to increase, these models demonstrate an enhanced ability to utilize distant information for tasks such as language modeling. This capability contrasts with human reading and writing habits, where it is uncommon to remember and use particularly distant information, except in cases of foreshadowing. In this paper, we aim to explore which kinds of words benefit more from long contexts in language models. By analyzing the changes in token probabilities with increasing context length, we find that content words (e.g., nouns, adjectives) and the initial tokens of words benefit the most. Frequent patterns in the context (N-grams) also significantly impact predictions. Additionally, the model's prior knowledge plays a crucial role in influencing predictions, especially for rare tokens. We also observe that language models become more confident with longer contexts, resulting in sharper probability distributions. This overconfidence may contribute to the increasing probabilities of tokens with distant contextual information. We hope that our analysis will help the community better understand long-text language modeling and contribute to the design of more reliable long-context models.