BERT-based distractor generation for Swedish reading comprehension questions using a small-scale dataset

TL;DR

This paper introduces a BERT-based approach for generating plausible distractors for Swedish reading comprehension MCQs using a small dataset, and evaluates its effectiveness from both student and teacher perspectives.

Contribution

It presents a novel BERT-based method for distractor generation in Swedish MCQs and releases a new small-scale dataset for training and evaluation.

Findings

Generated plausible distractors for over 50% of MCQs from students' view

Approximately 50% of distractors were deemed appropriate by teachers

Provides a methodology for assessing distractor quality

Abstract

An important part when constructing multiple-choice questions (MCQs) for reading comprehension assessment are the distractors, the incorrect but preferably plausible answer options. In this paper, we present a new BERT-based method for automatically generating distractors using only a small-scale dataset. We also release a new such dataset of Swedish MCQs (used for training the model), and propose a methodology for assessing the generated distractors. Evaluation shows that from a student's perspective, our method generated one or more plausible distractors for more than 50% of the MCQs in our test set. From a teacher's perspective, about 50% of the generated distractors were deemed appropriate. We also do a thorough analysis of the results.