Frustratingly Hard Evidence Retrieval for QA Over Books

TL;DR

This paper explores question answering over narrative books, highlighting the challenges of evidence retrieval and achieving state-of-the-art results on the NarrativeQA benchmark, while emphasizing the need for improved retrieval methods.

Contribution

It formulates BookQA as an open-domain QA task, applies state-of-the-art methods, and analyzes evidence retrieval difficulties specific to narrative books.

Findings

Achieved state-of-the-art on NarrativeQA benchmark

Evidence retrieval remains a significant challenge in BookQA

Highlights need for novel evidence retrieval solutions

Abstract

A lot of progress has been made to improve question answering (QA) in recent years, but the special problem of QA over narrative book stories has not been explored in-depth. We formulate BookQA as an open-domain QA task given its similar dependency on evidence retrieval. We further investigate how state-of-the-art open-domain QA approaches can help BookQA. Besides achieving state-of-the-art on the NarrativeQA benchmark, our study also reveals the difficulty of evidence retrieval in books with a wealth of experiments and analysis - which necessitates future effort on novel solutions for evidence retrieval in BookQA.