BookQA: Stories of Challenges and Opportunities

TL;DR

This paper introduces a BookQA system that leverages passage retrieval and memory networks, enhanced by BERT-based pretraining, to answer questions from full book texts, highlighting challenges and future research directions.

Contribution

The paper presents a novel BookQA approach combining passage retrieval, memory networks, and BERT-based pretraining, demonstrating significant improvements on the NarrativeQA dataset.

Findings

BERT-based retrieval and pretraining significantly improve performance.

NarrativeQA remains a highly challenging dataset for BookQA.

Current methods face bottlenecks in text representation, retrieval, and reasoning.

Abstract

We present a system for answering questions based on the full text of books (BookQA), which first selects book passages given a question at hand, and then uses a memory network to reason and predict an answer. To improve generalization, we pretrain our memory network using artificial questions generated from book sentences. We experiment with the recently published NarrativeQA corpus, on the subset of Who questions, which expect book characters as answers. We experimentally show that BERT-based retrieval and pretraining improve over baseline results significantly. At the same time, we confirm that NarrativeQA is a highly challenging data set, and that there is need for novel research in order to achieve high-precision BookQA results. We analyze some of the bottlenecks of the current approach, and we argue that more research is needed on text representation, retrieval of relevant passages, and reasoning, including commonsense knowledge.