Generative Students: Using LLM-Simulated Student Profiles to Support Question Item Evaluation

TL;DR

This paper introduces Generative Students, a novel approach using GPT-4 to simulate student profiles for evaluating question quality, showing high correlation with real student responses and aiding instructors in question improvement.

Contribution

The paper presents a new prompt architecture for simulating student profiles with LLMs, enabling effective question evaluation and quality enhancement.

Findings

Generative students produced logical, profile-aligned responses.

High correlation between generative and real student responses.

Overlap in difficult questions identified by both groups.

Abstract

Evaluating the quality of automatically generated question items has been a long standing challenge. In this paper, we leverage LLMs to simulate student profiles and generate responses to multiple-choice questions (MCQs). The generative students' responses to MCQs can further support question item evaluation. We propose Generative Students, a prompt architecture designed based on the KLI framework. A generative student profile is a function of the list of knowledge components the student has mastered, has confusion about or has no evidence of knowledge of. We instantiate the Generative Students concept on the subject domain of heuristic evaluation. We created 45 generative students using GPT-4 and had them respond to 20 MCQs. We found that the generative students produced logical and believable responses that were aligned with their profiles. We then compared the generative students' responses to real students' responses on the same set of MCQs and found a high correlation. Moreover, there was considerable overlap in the difficult questions identified by generative students and real students. A subsequent case study demonstrated that an instructor could improve question quality based on the signals provided by Generative Students.