Improving Automated Distractor Generation for Math Multiple-choice Questions with Overgenerate-and-rank

TL;DR

This paper introduces an overgenerate-and-rank method to improve automatic distractor generation for math MCQs, aligning machine-generated options more closely with human-created distractors and enhancing their quality.

Contribution

The paper presents a novel overgenerate-and-rank approach that trains a ranking model to produce higher-quality distractors for math multiple-choice questions.

Findings

Ranking model improves alignment with human distractors

Generated distractors are more relevant after ranking

Human-authored distractors remain preferred over generated ones

Abstract

Multiple-choice questions (MCQs) are commonly used across all levels of math education since they can be deployed and graded at a large scale. A critical component of MCQs is the distractors, i.e., incorrect answers crafted to reflect student errors or misconceptions. Automatically generating them in math MCQs, e.g., with large language models, has been challenging. In this work, we propose a novel method to enhance the quality of generated distractors through overgenerate-and-rank, training a ranking model to predict how likely distractors are to be selected by real students. Experimental results on a real-world dataset and human evaluation with math teachers show that our ranking model increases alignment with human-authored distractors, although human-authored ones are still preferred over generated ones.