Methodological Foundations for AI-Driven Survey Question Generation

TL;DR

This paper develops a framework for using generative AI to create, test, and refine survey questions in educational research, emphasizing ethical considerations and interaction analysis.

Contribution

It introduces the Synthetic Question-Response Analysis (SQRA) framework for iterative prompt testing and applies Activity Theory to analyze AI-mediated engagement.

Findings

AI-generated questions show promising engagement but require validation.

Prompt engineering is crucial for reliable AI survey tools.

Analysis reveals both strengths and limitations of AI in survey contexts.

Abstract

This paper presents a methodological framework for using generative AI in educational survey research. We explore how Large Language Models (LLMs) can generate adaptive, context-aware survey questions and introduce the Synthetic Question-Response Analysis (SQRA) framework, which enables iterative testing and refinement of AI-generated prompts prior to deployment with human participants. Guided by Activity Theory, we analyze how AI tools mediate participant engagement and learning, and we examine ethical issues such as bias, privacy, and transparency. Through sentiment, lexical, and structural analyses of both AI-to-AI and AI-to-human survey interactions, we evaluate the alignment and effectiveness of these questions. Our findings highlight the promise and limitations of AI-driven survey instruments, emphasizing the need for robust prompt engineering and validation to support trustworthy, scalable, and contextually relevant data collection in engineering education.