Psychometric Item Validation Using Virtual Respondents with Trait-Response Mediators

TL;DR

This paper introduces a scalable framework using LLMs to simulate virtual respondents with mediators for validating psychometric survey items, reducing reliance on costly human data collection.

Contribution

It presents a novel method for virtual respondent simulation that improves construct validity assessment of psychometric items using LLM-generated mediators.

Findings

Effective identification of high-validity items across three trait theories

LLMs can generate plausible mediators from trait definitions

Simulation framework reduces need for large-scale human data

Abstract

As psychometric surveys are increasingly used to assess the traits of large language models (LLMs), the need for scalable survey item generation suited for LLMs has also grown. A critical challenge here is ensuring the construct validity of generated items, i.e., whether they truly measure the intended trait. Traditionally, this requires costly, large-scale human data collection. To make it efficient, we present a framework for virtual respondent simulation using LLMs. Our central idea is to account for mediators: factors through which the same trait can give rise to varying responses to a survey item. By simulating respondents with diverse mediators, we identify survey items that robustly measure intended traits. Experiments on three psychological trait theories (Big5, Schwartz, VIA) show that our mediator generation methods and simulation framework effectively identify high-validity items. LLMs demonstrate the ability to generate plausible mediators from trait definitions and to simulate respondent behavior for item validation. Our problem formulation, metrics, methodology, and dataset open a new direction for cost-effective survey development and a deeper understanding of how LLMs simulate human survey responses. We publicly release our dataset and code to support future work.