Simple and Effective Semi-Supervised Question Answering

TL;DR

This paper presents a semi-supervised approach for extractive question answering that leverages document structure and cloze-style questions to achieve high performance with minimal labeled data, reducing the need for large annotated corpora.

Contribution

The authors introduce a novel semi-supervised QA system that uses document structure to generate cloze questions and pre-trains neural networks, enabling effective QA with very few labeled examples.

Findings

Achieves over 50% F1 on SQuAD and TriviaQA with less than 1,000 labeled examples.

Creates and releases 3.2 million cloze-style questions for QA system development.

Demonstrates high effectiveness across diverse datasets with minimal supervision.

Abstract

Recent success of deep learning models for the task of extractive Question Answering (QA) is hinged on the availability of large annotated corpora. However, large domain specific annotated corpora are limited and expensive to construct. In this work, we envision a system where the end user specifies a set of base documents and only a few labelled examples. Our system exploits the document structure to create cloze-style questions from these base documents; pre-trains a powerful neural network on the cloze style questions; and further fine-tunes the model on the labeled examples. We evaluate our proposed system across three diverse datasets from different domains, and find it to be highly effective with very little labeled data. We attain more than 50% F1 score on SQuAD and TriviaQA with less than a thousand labelled examples. We are also releasing a set of 3.2M cloze-style questions for practitioners to use while building QA systems.