Zero-Shot Open-Book Question Answering

TL;DR

This paper presents a zero-shot open-book question answering system for AWS documentation, combining retrieval and extraction components, trained on general QA datasets, achieving notable accuracy without domain-specific data.

Contribution

It introduces a novel zero-shot QA approach for technical documents, with a new real-world dataset and a two-step retrieval and extraction architecture.

Findings

Achieved 49% F1 and 39% EM scores end-to-end without domain-specific training.

Developed a new dataset based on real AWS customer questions.

Demonstrated effectiveness of combining retrieval and extraction in zero-shot setting.

Abstract

Open book question answering is a subset of question answering tasks where the system aims to find answers in a given set of documents (open-book) and common knowledge about a topic. This article proposes a solution for answering natural language questions from a corpus of Amazon Web Services (AWS) technical documents with no domain-specific labeled data (zero-shot). These questions can have yes-no-none answers, short answers, long answers, or any combination of the above. This solution comprises a two-step architecture in which a retriever finds the right document and an extractor finds the answers in the retrieved document. We are introducing a new test dataset for open-book QA based on real customer questions on AWS technical documentation. After experimenting with several information retrieval systems and extractor models based on extractive language models, the solution attempts to find the yes-no-none answers and text answers in the same pass. The model is trained on the The Stanford Question Answering Dataset - SQuAD (Rajpurkaret al., 2016) and Natural Questions (Kwiatkowski et al., 2019) datasets. We were able to achieve 49% F1 and 39% exact match score (EM) end-to-end with no domain-specific training.