Using the Hammer Only on Nails: A Hybrid Method for Evidence Retrieval for Question Answering

TL;DR

This paper presents a hybrid evidence retrieval method for question answering that combines traditional IR and neural approaches, improving accuracy and speed by intelligently routing questions to the most suitable method.

Contribution

A novel hybrid evidence retrieval approach using a simple routing classifier that outperforms individual methods in accuracy and efficiency on multiple QA datasets.

Findings

Hybrid method outperforms individual IR and neural approaches.

Routing classifier improves retrieval accuracy.

Significant speedup over neural methods.

Abstract

Evidence retrieval is a key component of explainable question answering (QA). We argue that, despite recent progress, transformer network-based approaches such as universal sentence encoder (USE-QA) do not always outperform traditional information retrieval (IR) methods such as BM25 for evidence retrieval for QA. We introduce a lexical probing task that validates this observation: we demonstrate that neural IR methods have the capacity to capture lexical differences between questions and answers, but miss obvious lexical overlap signal. Learning from this probing analysis, we introduce a hybrid approach for evidence retrieval that combines the advantages of both IR directions. Our approach uses a routing classifier that learns when to direct incoming questions to BM25 vs. USE-QA for evidence retrieval using very simple statistics, which can be efficiently extracted from the top candidate evidence sentences produced by a BM25 model. We demonstrate that this hybrid evidence retrieval generally performs better than either individual retrieval strategy on three QA datasets: OpenBookQA, ReQA SQuAD, and ReQA NQ. Furthermore, we show that the proposed routing strategy is considerably faster than neural methods, with a runtime that is up to 5 times faster than USE-QA.