From traces to measures: Large language models as a tool for psychological measurement from text

TL;DR

This paper presents a workflow for developing and evaluating large language model-based measures of psychological features from text, addressing psychometric reliability and validity considerations.

Contribution

It introduces a systematic approach for creating and assessing LLM-based psychological measures, including an example measuring attitude certainty, importance, and moralization.

Findings

Good internal consistency of measures

Partial validity according to criteria

Workflow aids in psychometric evaluation of LLM-based measures

Abstract

Large language models are increasingly being used to label or rate psychological features in text data. This approach helps address one of the limiting factors of digital trace data - their lack of an inherent target of measurement. However, this approach is also a form of psychological measurement (using observable variables to quantify a hypothetical latent construct). As such, these ratings are subject to the same psychometric considerations of reliability and validity as more standard psychological measures. Here we present a workflow for developing and evaluating large language model based measures of psychological features which incorporate these considerations. We also provide an example, attempting to measure the previously established constructs of attitude certainty, importance and moralization from text. Using a pool of prompts adapted from existing measurement instruments, we find they have good levels of internal consistency but only partially meet validity criteria.