Knowing More About Questions Can Help: Improving Calibration in Question Answering

TL;DR

This paper introduces a new calibration method for question answering that uses input information and data augmentation, significantly improving calibration accuracy across various models and settings.

Contribution

It presents a novel calibration approach that incorporates input features and demonstrates its effectiveness in both reading comprehension and open retrieval tasks.

Findings

5-10% calibration accuracy improvements

Effective across multiple models and tasks

First calibration study in open retrieval setting

Abstract

We study calibration in question answering, estimating whether model correctly predicts answer for each question. Unlike prior work which mainly rely on the model's confidence score, our calibrator incorporates information about the input example (e.g., question and the evidence context). Together with data augmentation via back translation, our simple approach achieves 5-10% gains in calibration accuracy on reading comprehension benchmarks. Furthermore, we present the first calibration study in the open retrieval setting, comparing the calibration accuracy of retrieval-based span prediction models and answer generation models. Here again, our approach shows consistent gains over calibrators relying on the model confidence. Our simple and efficient calibrator can be easily adapted to many tasks and model architectures, showing robust gains in all settings.