Scalable Generation and Validation of Isomorphic Physics Problems with GenAI

TL;DR

This paper introduces a scalable framework using Generative AI to create and validate large banks of isomorphic physics problems for asynchronous assessments, ensuring consistent difficulty and diverse contexts.

Contribution

The work presents a novel prompt chaining approach for generating structurally varied physics problems and evaluates their difficulty using open-source language models and student data.

Findings

73% of generated problem banks are statistically homogeneous in difficulty.

Language models' performance correlates with student performance (up to 0.594 Pearson's rho).

Model scale impacts validation effectiveness, with mid-sized models being optimal.

Abstract

Traditional synchronous STEM assessments face growing challenges including accessibility barriers, security concerns from resource-sharing platforms, and limited comparability across institutions. We present a framework for generating and evaluating large-scale isomorphic physics problem banks using Generative AI to enable asynchronous, multi-attempt assessments. Isomorphic problems test identical concepts through varied surface features and contexts, providing richer variation than conventional parameterized questions while maintaining consistent difficulty. Our generation framework employs prompt chaining and tool use to achieve precise control over structural variations (numeric values, spatial relations) alongside diverse contextual variations. For pre-deployment validation, we evaluate generated items using 17 open-source language models (LMs) (0.6B-32B) and compare against actual student performance (N>200) across three midterm exams. Results show that 73% of deployed banks achieve statistically homogeneous difficulty, and LMs pattern correlate strongly with student performance (Pearson's $\rho$ up to 0.594). Additionally, LMs successfully identify problematic variants, such as ambiguous problem texts. Model scale also proves critical for effective validation, where extremely small (<4B) and large (>14B) models exhibit floor and ceiling effects respectively, making mid-sized models optimal for detecting difficulty outliers.