On the scaling relationship between cloze probabilities and language model next-token prediction

TL;DR

This paper investigates how larger language models better predict human-like responses in cloze tasks by balancing semantic understanding and sensitivity to lexical details, revealing their improved predictive capabilities.

Contribution

It demonstrates that larger models assign higher-quality next-token probabilities and are more semantically aligned with human responses, highlighting the role of memorization capacity.

Findings

Larger models better predict human cloze responses.

They are less sensitive to lexical co-occurrence.

They assign higher-quality next-token probabilities.

Abstract

Recent work has shown that larger language models have better predictive power for eye movement and reading time data. While even the best models under-allocate probability mass to human responses, larger models assign higher-quality estimates of next tokens and their likelihood of production in cloze data because they are less sensitive to lexical co-occurrence statistics while being better aligned semantically to human cloze responses. The results provide support for the claim that the greater memorization capacity of larger models helps them guess more semantically appropriate words, but makes them less sensitive to low-level information that is relevant for word recognition.