Crowdsourcing Multiple Choice Science Questions

TL;DR

This paper introduces a new crowdsourcing method for creating high-quality, domain-specific multiple choice science questions by combining large text corpora and existing questions, resulting in the SciQ dataset and improved exam accuracy.

Contribution

The paper presents a novel crowdsourcing approach that leverages domain text and existing questions to generate high-quality science questions, along with the creation of the SciQ dataset.

Findings

The method produces questions indistinguishable from original ones by humans.

Adding SciQ data improves accuracy on real science exams.

The approach effectively generates diverse, relevant science questions.

Abstract

We present a novel method for obtaining high-quality, domain-targeted multiple choice questions from crowd workers. Generating these questions can be difficult without trading away originality, relevance or diversity in the answer options. Our method addresses these problems by leveraging a large corpus of domain-specific text and a small set of existing questions. It produces model suggestions for document selection and answer distractor choice which aid the human question generation process. With this method we have assembled SciQ, a dataset of 13.7K multiple choice science exam questions (Dataset available at http://allenai.org/data.html). We demonstrate that the method produces in-domain questions by providing an analysis of this new dataset and by showing that humans cannot distinguish the crowdsourced questions from original questions. When using SciQ as additional training data to existing questions, we observe accuracy improvements on real science exams.