From Prompts to Constructs: A Dual-Validity Framework for LLM Research in Psychology

TL;DR

This paper proposes a dual-validity framework for LLM research in psychology, emphasizing the need for rigorous validation standards to distinguish genuine psychological phenomena from statistical artifacts.

Contribution

It introduces a dual-validity framework that guides the appropriate validation of LLM claims in psychology, integrating measurement reliability and causal inference standards.

Findings

Current practices often treat pattern matching as evidence of psychological phenomena.

Different claims about LLMs require different validation strategies.

Moving towards computational constructs and scalable evidence standards is necessary.

Abstract

Large language models (LLMs) are rapidly being adopted across psychology, serving as research tools, experimental subjects, human simulators, and computational models of cognition. However, the application of human measurement tools to these systems can produce contradictory results, raising concerns that many findings are measurement phantoms--statistical artifacts rather than genuine psychological phenomena. In this Perspective, we argue that building a robust science of AI psychology requires integrating two of our field's foundational pillars: the principles of reliable measurement and the standards for sound causal inference. We present a dual-validity framework to guide this integration, which clarifies how the evidence needed to support a claim scales with its scientific ambition. Using an LLM to classify text may require only basic accuracy checks, whereas claiming it can simulate anxiety demands a far more rigorous validation process. Current practice systematically fails to meet these requirements, often treating statistical pattern matching as evidence of psychological phenomena. The same model output--endorsing "I am anxious"--requires different validation strategies depending on whether researchers claim to measure, characterize, simulate, or model psychological constructs. Moving forward requires developing computational analogues of psychological constructs and establishing clear, scalable standards of evidence rather than the uncritical application of human measurement tools.