Human- vs. AI-generated tests: dimensionality and information accuracy in latent trait evaluation

TL;DR

This study compares AI-generated and human-developed questionnaires to evaluate differences in dimensionality and information accuracy, highlighting the importance of statistical validation for AI tools in social research.

Contribution

It introduces a method to assess psychometric properties of AI-generated measurement instruments using Bayesian models, revealing key differences from human instruments.

Findings

AI questionnaires showed similar wording but differed in dimensionality.

Differences in item and test information distribution across traits.

Statistical validation is crucial for AI-driven measurement tools.

Abstract

Artificial Intelligence (AI) and large language models (LLMs) are increasingly used in social and psychological research. Among potential applications, LLMs can be used to generate, customise, or adapt measurement instruments. This study presents a preliminary investigation of AI-generated questionnaires by comparing two ChatGPT-based adaptations of the Body Awareness Questionnaire (BAQ) with the validated human-developed version. The AI instruments were designed with different levels of explicitness in content and instructions on construct facets, and their psychometric properties were assessed using a Bayesian Graded Response Model. Results show that although surface wording between AI and original items was similar, differences emerged in dimensionality and in the distribution of item and test information across latent traits. These findings illustrate the importance of applying statistical measures of accuracy to ensure the validity and interpretability of AI-driven tools.