Does Multiple Choice Have a Future in the Age of Generative AI? A Posttest-only RCT

TL;DR

This study compares multiple-choice questions and open-response tasks in learning, finding MCQs as effective and more time-efficient, with GPT models aiding in grading, thus questioning the future dominance of open responses.

Contribution

It provides empirical evidence on MCQ effectiveness relative to open responses and introduces GPT-based autograding methods for open responses.

Findings

No significant difference in learning outcomes across conditions.

MCQ condition required less completion time.

GPT models effectively autograded open responses for low-stakes assessment.

Abstract

The role of multiple-choice questions (MCQs) as effective learning tools has been debated in past research. While MCQs are widely used due to their ease in grading, open response questions are increasingly used for instruction, given advances in large language models (LLMs) for automated grading. This study evaluates MCQs effectiveness relative to open-response questions, both individually and in combination, on learning. These activities are embedded within six tutor lessons on advocacy. Using a posttest-only randomized control design, we compare the performance of 234 tutors (790 lesson completions) across three conditions: MCQ only, open response only, and a combination of both. We find no significant learning differences across conditions at posttest, but tutors in the MCQ condition took significantly less time to complete instruction. These findings suggest that MCQs are as effective, and more efficient, than open response tasks for learning when practice time is limited. To further enhance efficiency, we autograded open responses using GPT-4o and GPT-4-turbo. GPT models demonstrate proficiency for purposes of low-stakes assessment, though further research is needed for broader use. This study contributes a dataset of lesson log data, human annotation rubrics, and LLM prompts to promote transparency and reproducibility.