Using AI to replicate human experimental results: a motion study

TL;DR

This study demonstrates that large language models like GPT-4 can reliably replicate human judgments in psycholinguistic experiments, suggesting their potential as tools for expanding linguistic research without losing interpretative validity.

Contribution

The paper provides empirical evidence that LLMs can accurately mirror human responses in linguistic experiments, supporting their use as reliable tools in psycholinguistic research.

Findings

Strong correlations (Spearman's rho = .73-.96) between human and AI responses.

LLMs can replicate nuanced human judgments in linguistic tasks.

AI responses generally do not alter overall interpretative outcomes.

Abstract

This paper explores the potential of large language models (LLMs) as reliable analytical tools in linguistic research, focusing on the emergence of affective meanings in temporal expressions involving manner-of-motion verbs. While LLMs like GPT-4 have shown promise across a range of tasks, their ability to replicate nuanced human judgements remains under scrutiny. We conducted four psycholinguistic studies (on emergent meanings, valence shifts, verb choice in emotional contexts, and sentence-emoji associations) first with human participants and then replicated the same tasks using an LLM. Results across all studies show a striking convergence between human and AI responses, with statistical analyses (e.g., Spearman's rho = .73-.96) indicating strong correlations in both rating patterns and categorical choices. While minor divergences were observed in some cases, these did not alter the overall interpretative outcomes. These findings offer compelling evidence that LLMs can augment traditional human-based experimentation, enabling broader-scale studies without compromising interpretative validity. This convergence not only strengthens the empirical foundation of prior human-based findings but also opens possibilities for hypothesis generation and data expansion through AI. Ultimately, our study supports the use of LLMs as credible and informative collaborators in linguistic inquiry.