Surprisal and Metaphor Novelty Judgments: Moderate Correlations and Divergent Scaling Effects Revealed by Corpus-Based and Synthetic Datasets

TL;DR

This study explores how surprisal, a measure of predictability in language models, relates to metaphor novelty judgments, revealing moderate correlations and contrasting scaling effects across datasets, thus highlighting surprisal's partial relevance to linguistic creativity.

Contribution

It introduces a corpus-based and synthetic dataset analysis of surprisal's correlation with metaphor novelty, revealing divergent scaling patterns and limitations of surprisal as a creativity metric.

Findings

Moderate correlation between surprisal and metaphor novelty scores.

Inverse scaling effect on corpus-based data with increasing model size.

Positive scaling effect on synthetic data with increasing model quality.

Abstract

Novel metaphor comprehension involves complex semantic processes and linguistic creativity, making it an interesting task for studying language models (LMs). This study investigates whether surprisal, a probabilistic measure of predictability in LMs, correlates with annotations of metaphor novelty in different datasets. We analyse the surprisal of metaphoric words in corpus-based and synthetic metaphor datasets using 16 causal LM variants. We propose a cloze-style surprisal method that conditions on full-sentence context. Results show that LM surprisal yields significant moderate correlations with scores/labels of metaphor novelty. We further identify divergent scaling patterns: on corpus-based data, correlation strength decreases with model size (inverse scaling effect), whereas on synthetic data it increases (quality-power hypothesis). We conclude that while surprisal can partially account for annotations of metaphor novelty, it remains limited as a metric of linguistic creativity. Code and data are publicly available: https://github.com/OmarMomen14/surprisal-metaphor-novelty