Assessing Distractors in Multiple-Choice Tests

TL;DR

This paper introduces automated metrics to evaluate distractors in multiple-choice reading comprehension tests, focusing on incorrectness, plausibility, and diversity to improve assessment quality.

Contribution

It proposes novel automated assessment metrics for distractor quality, integrating classification, probability, and embedding-based diversity measures.

Findings

The proposed metrics effectively evaluate distractor quality.

Plausibility correlates with ChatGPT's interpretations.

Diversity assessment improves distractor selection.

Abstract

Multiple-choice tests are a common approach for assessing candidates' comprehension skills. Standard multiple-choice reading comprehension exams require candidates to select the correct answer option from a discrete set based on a question in relation to a contextual passage. For appropriate assessment, the distractor answer options must by definition be incorrect but plausible and diverse. However, generating good quality distractors satisfying these criteria is a challenging task for content creators. We propose automated assessment metrics for the quality of distractors in multiple-choice reading comprehension tests. Specifically, we define quality in terms of the incorrectness, plausibility and diversity of the distractor options. We assess incorrectness using the classification ability of a binary multiple-choice reading comprehension system. Plausibility is assessed by considering the distractor confidence - the probability mass associated with the distractor options for a standard multi-class multiple-choice reading comprehension system. Diversity is assessed by pairwise comparison of an embedding-based equivalence metric between the distractors of a question. To further validate the plausibility metric we compare against candidate distributions over multiple-choice questions and agreement with a ChatGPT model's interpretation of distractor plausibility and diversity.