Improving Retrieval-Based Question Answering with Deep Inference Models

TL;DR

This paper introduces a two-step deep inference approach that combines information retrieval and deep learning to improve multi-choice science question answering, achieving over 3% accuracy gain.

Contribution

It presents a novel combination of retrieval models and deep inference architectures for complex science question answering, outperforming previous retrieval-only methods.

Findings

Over 3% absolute accuracy improvement over retrieval-based baseline

Effective decomposition of question answering into retrieval and inference sub-tasks

Neural network ensemble improves answer prediction accuracy

Abstract

Question answering is one of the most important and difficult applications at the border of information retrieval and natural language processing, especially when we talk about complex science questions which require some form of inference to determine the correct answer. In this paper, we present a two-step method that combines information retrieval techniques optimized for question answering with deep learning models for natural language inference in order to tackle the multi-choice question answering in the science domain. For each question-answer pair, we use standard retrieval-based models to find relevant candidate contexts and decompose the main problem into two different sub-problems. First, assign correctness scores for each candidate answer based on the context using retrieval models from Lucene. Second, we use deep learning architectures to compute if a candidate answer can be inferred from some well-chosen context consisting of sentences retrieved from the knowledge base. In the end, all these solvers are combined using a simple neural network to predict the correct answer. This proposed two-step model outperforms the best retrieval-based solver by over 3% in absolute accuracy.