Context Generation Improves Open Domain Question Answering

TL;DR

This paper introduces a two-stage context generation approach for closed-book question answering, significantly improving performance by generating relevant contexts from pretrained language models without extra training.

Contribution

Proposes a novel coarse-to-fine framework that generates and marginalizes over contexts, enhancing closed-book QA performance without additional parameters or finetuning.

Findings

Outperforms previous closed-book QA methods in accuracy.

Matches open-book methods that use external knowledge.

Does not require extra training or parameters.

Abstract

Closed-book question answering (QA) requires a model to directly answer an open-domain question without access to any external knowledge. Prior work on closed-book QA either directly finetunes or prompts a pretrained language model (LM) to leverage the stored knowledge. However, they do not fully exploit the parameterized knowledge. To address this issue, we propose a two-stage, closed-book QA framework which employs a coarse-to-fine approach to extract relevant knowledge and answer a question. Our approach first generates a related context for a given question by prompting a pretrained LM. We then prompt the same LM for answer prediction using the generated context and the question. Additionally, to eliminate failure caused by context uncertainty, we marginalize over generated contexts. Experimental results on three QA benchmarks show that our method significantly outperforms previous closed-book QA methods (e.g. exact matching 68.6% vs. 55.3%), and is on par with open-book methods that exploit external knowledge sources (e.g. 68.6% vs. 68.0%). Our method is able to better exploit the stored knowledge in pretrained LMs without adding extra learnable parameters or needing finetuning, and paves the way for hybrid models that integrate pretrained LMs with external knowledge.