Towards Confident Machine Reading Comprehension

TL;DR

This paper introduces Mr.C, a confidence estimation model for machine reading comprehension that improves the system's ability to avoid incorrect answers and better predict answerability, especially under domain shift conditions.

Contribution

The paper presents a novel confidence estimation approach using gradient-based features, enhancing answer correctness and answerability predictions in MRC systems.

Findings

Mr.C improves AUC scores by up to 4 points.

Mr.C enhances answerability prediction by 5 F1 points.

Gradient-based features contribute significantly to confidence estimation.

Abstract

There has been considerable progress on academic benchmarks for the Reading Comprehension (RC) task with State-of-the-Art models closing the gap with human performance on extractive question answering. Datasets such as SQuAD 2.0 & NQ have also introduced an auxiliary task requiring models to predict when a question has no answer in the text. However, in production settings, it is also necessary to provide confidence estimates for the performance of the underlying RC model at both answer extraction and "answerability" detection. We propose a novel post-prediction confidence estimation model, which we call Mr.C (short for Mr. Confident), that can be trained to improve a system's ability to refrain from making incorrect predictions with improvements of up to 4 points as measured by Area Under the Curve (AUC) scores. Mr.C can benefit from a novel white-box feature that leverages the underlying RC model's gradients. Performance prediction is particularly important in cases of domain shift (as measured by training RC models on SQUAD 2.0 and evaluating on NQ), where Mr.C not only improves AUC, but also traditional answerability prediction (as measured by a 5 point improvement in F1).