Automatic Item Generation for Personality Situational Judgment Tests with Large Language Models

TL;DR

This study develops and evaluates an automated framework using GPT-4 and ChatGPT-5 for generating personality situational judgment tests, demonstrating high reliability, validity, and reproducibility, thus reducing development time and reliance on experts.

Contribution

The paper introduces a structured, generalizable LLM-based approach for automatic item generation of personality SJTs, improving efficiency and quality over traditional methods.

Findings

Optimized prompts and temperature of 1.0 yield best content validity.

Approach produces reproducible, high-quality items across models.

Generated SJTs show satisfactory reliability and validity for most traits.

Abstract

Personality assessment through situational judgment tests (SJTs) offers unique advantages over traditional Likert-type self-report scales, yet their development remains labor-intensive, time-consuming, and heavily dependent on subject matter experts. Recent advances in large language models (LLMs) have shown promise for automatic item generation (AIG). Building on these developments, the present study focuses on developing and evaluating a structured and generalizable framework for automatically generating personality SJTs, using GPT-4 and ChatGPT-5 as empirical examples. Three studies were conducted. Study 1 systematically compared the effects of prompt design and temperature settings on the content validity of LLM-generated items to develop an effective and stable LLM-based AIG approach for personality SJT. Results showed that optimized prompts and a temperature of 1.0 achieved the best balance of creativity and accuracy on GPT-4. Study 2 examined the cross-model generalizability and reproducibility of this automated SJT generation approach through multiple rounds. The results showed that the approach consistently produced reproducible and high-quality items on ChatGPT-5. Study 3 evaluated the psychometric properties of LLM-generated SJTs covering five facets of the Big Five personality traits. Results demonstrated satisfactory reliability and validity across most facets, though limitations were observed in the convergent validity of the compliance facet and certain aspects of criterion-related validity. These findings provide robust evidence that the proposed LLM-based AIG approach can produce culturally appropriate and psychometrically sound SJTs with efficiency comparable to or exceeding traditional methods.