Can LLMs Model Incorrect Student Reasoning? A Case Study on Distractor Generation

TL;DR

This paper investigates how large language models generate plausible student misconceptions for multiple-choice distractors, revealing their reasoning process, common failure modes, and the impact of providing correct solutions in prompts.

Contribution

It introduces a taxonomy for analyzing LLM reasoning in distractor generation, compares their strategies to learning science best practices, and evaluates how prompt design affects output quality.

Findings

LLMs typically solve the problem first, then generate misconceptions.

Errors mainly occur in recovering solutions and selecting distractors.

Including the correct solution in prompts improves distractor quality by 8%.

Abstract

Modeling plausible student misconceptions is critical for AI in education. In this work, we examine how large language models (LLMs) reason about misconceptions when generating multiple-choice distractors, a task that requires modeling incorrect yet plausible answers by coordinating solution knowledge, simulating student misconceptions, and evaluating plausibility. We introduce a taxonomy for analyzing the strategies used by state-of-the-art LLMs, examining their reasoning procedures and comparing them to established best practices in the learning sciences. Our structured analysis reveals a surprising alignment between their processes and best practices: the models typically solve the problem correctly first, then articulate and simulate multiple potential misconceptions, and finally select a set of distractors. An analysis of failure modes reveals that errors arise primarily from failures in recovering the correct solution and selecting among response candidates, rather than simulating errors or structuring the process. Consistent with these results, we find that providing the correct solution in the prompt improves alignment with human-authored distractors by 8%, highlighting the critical role of anchoring to the correct solution when generating plausible incorrect student reasoning. Overall, our analysis offers a structured and interpretable lens into LLMs' ability to model incorrect student reasoning and produce high-quality distractors.