Can GPT replace human raters? Validity and reliability of machine-generated norms for metaphors

TL;DR

This study evaluates the validity and reliability of GPT-generated ratings for metaphors in terms of familiarity, comprehensibility, and imageability, comparing them to human ratings and behavioral responses, across English and Italian datasets.

Contribution

It provides the first comprehensive validation of GPT models for rating complex metaphors, demonstrating their potential to replace or augment human raters in psycholinguistic research.

Findings

GPT ratings correlate with human ratings, especially in familiarity and imageability.

Larger GPT models outperform smaller ones in rating accuracy.

GPT ratings predict behavioral and electrophysiological responses effectively.

Abstract

As Large Language Models (LLMs) are increasingly being used in scientific research, the issue of their trustworthiness becomes crucial. In psycholinguistics, LLMs have been recently employed in automatically augmenting human-rated datasets, with promising results obtained by generating ratings for single words. Yet, performance for ratings of complex items, i.e., metaphors, is still unexplored. Here, we present the first assessment of the validity and reliability of ratings of metaphors on familiarity, comprehensibility, and imageability, generated by three GPT models for a total of 687 items gathered from the Italian Figurative Archive and three English studies. We performed a thorough validation in terms of both alignment with human data and ability to predict behavioral and electrophysiological responses. We found that machine-generated ratings positively correlated with human-generated ones. Familiarity ratings reached moderate-to-strong correlations for both English and Italian metaphors, although correlations weakened for metaphors with high sensorimotor load. Imageability showed moderate correlations in English and moderate-to-strong in Italian. Comprehensibility for English metaphors exhibited the strongest correlations. Overall, larger models outperformed smaller ones and greater human-model misalignment emerged with familiarity and imageability. Machine-generated ratings significantly predicted response times and the EEG amplitude, with a strength comparable to human ratings. Moreover, GPT ratings obtained across independent sessions were highly stable. We conclude that GPT, especially larger models, can validly and reliably replace - or augment - human subjects in rating metaphor properties. Yet, LLMs align worse with humans when dealing with conventionality and multimodal aspects of metaphorical meaning, calling for careful consideration of the nature of stimuli.