The creative psychometric item generator: a framework for item generation and validation using large language models

TL;DR

This paper introduces CPIG, a framework using large language models to generate and validate creative problem-solving test items, demonstrating their effectiveness in creating valid and reliable creativity assessments.

Contribution

The paper presents a novel psychometrically inspired framework, CPIG, that leverages LLMs for automated creation and validation of creativity test items.

Findings

CPIG generates valid and reliable creativity test items.

Items from later iterations elicit more creative responses.

The effect is not due to known biases in evaluation.

Abstract

Increasingly, large language models (LLMs) are being used to automate workplace processes requiring a high degree of creativity. While much prior work has examined the creativity of LLMs, there has been little research on whether they can generate valid creativity assessments for humans despite the increasingly central role of creativity in modern economies. We develop a psychometrically inspired framework for creating test items (questions) for a classic free-response creativity test: the creative problem-solving (CPS) task. Our framework, the creative psychometric item generator (CPIG), uses a mixture of LLM-based item generators and evaluators to iteratively develop new prompts for writing CPS items, such that items from later iterations will elicit more creative responses from test takers. We find strong empirical evidence that CPIG generates valid and reliable items and that this effect is not attributable to known biases in the evaluation process. Our findings have implications for employing LLMs to automatically generate valid and reliable creativity tests for humans and AI.