The Frequency Confound in Language-Model Surprisal and Metaphor Novelty

TL;DR

This paper investigates how lexical frequency, rather than language-model surprisal, primarily influences metaphor novelty judgments, challenging prior assumptions about surprisal's role.

Contribution

It demonstrates that word frequency is a stronger predictor of metaphor novelty than surprisal across different models and training stages.

Findings

Word frequency outperforms surprisal in predicting metaphor novelty.

Surprisal's correlation with novelty peaks early and then diminishes.

Lexical frequency may be the main factor behind previous surprisal-based findings.

Abstract

Language-model (LM) surprisal is widely used as a proxy for contextual predictability and has been reported to correlate with metaphor novelty judgments. However, surprisal is tightly intertwined with lexical frequency. We explore this interaction on metaphor novelty ratings using two different word frequency measures. We analyse surprisal estimates from eight Pythia model sizes and 154 training checkpoints. Across settings, word frequency is a stronger predictor of metaphor novelty than surprisal. Across training stages, the surprisal--novelty association peaks at an early stage and then falls again, mirroring a similarly timed increase in the surprisal--frequency association. These results suggest that the often-reported optimal LM surprisal settings may incorrectly associate contextual predictability with metaphor novelty and processing difficulty, whereas lexical frequency may be the major underlying factor.