ChatGPTest: opportunities and cautionary tales of utilizing AI for questionnaire pretesting

TL;DR

This paper investigates the potential of GPT models to assist in pretesting survey questionnaires, proposing a new AI-assisted stage to improve survey design efficiency while emphasizing the importance of human judgment.

Contribution

It introduces a novel application of GPT models for questionnaire pretesting, highlighting how AI can complement traditional methods in survey research.

Findings

GPT feedback can reduce the number of iterative revisions

AI-assisted pretesting enhances early survey design stages

Researchers' judgment remains crucial in interpreting AI feedback

Abstract

The rapid advancements in generative artificial intelligence have opened up new avenues for enhancing various aspects of research, including the design and evaluation of survey questionnaires. However, the recent pioneering applications have not considered questionnaire pretesting. This article explores the use of GPT models as a useful tool for pretesting survey questionnaires, particularly in the early stages of survey design. Illustrated with two applications, the article suggests incorporating GPT feedback as an additional stage before human pretesting, potentially reducing successive iterations. The article also emphasizes the indispensable role of researchers' judgment in interpreting and implementing AI-generated feedback.