Incremental Improvement of a Question Answering System by Re-ranking Answer Candidates using Machine Learning

TL;DR

This paper presents a re-ranking method using machine learning to improve answer selection in a question answering system without retraining the entire model, demonstrating a 9.15% boost in mean reciprocal rank.

Contribution

It introduces a novel re-ranking approach for QA systems that enhances answer accuracy using n-gram features and gradient boosting, applicable post-deployment.

Findings

Re-ranking improves top-n accuracy and mean reciprocal rank.

The approach benefits deployed QA systems without retraining.

Achieved a 9.15% increase in mean reciprocal rank.

Abstract

We implement a method for re-ranking top-10 results of a state-of-the-art question answering (QA) system. The goal of our re-ranking approach is to improve the answer selection given the user question and the top-10 candidates. We focus on improving deployed QA systems that do not allow re-training or re-training comes at a high cost. Our re-ranking approach learns a similarity function using n-gram based features using the query, the answer and the initial system confidence as input. Our contributions are: (1) we generate a QA training corpus starting from 877 answers from the customer care domain of T-Mobile Austria, (2) we implement a state-of-the-art QA pipeline using neural sentence embeddings that encode queries in the same space than the answer index, and (3) we evaluate the QA pipeline and our re-ranking approach using a separately provided test set. The test set can be considered to be available after deployment of the system, e.g., based on feedback of users. Our results show that the system performance, in terms of top-n accuracy and the mean reciprocal rank, benefits from re-ranking using gradient boosted regression trees. On average, the mean reciprocal rank improves by 9.15%.