Using calibrator to improve robustness in Machine Reading Comprehension

TL;DR

This paper introduces a post-hoc calibrator based on XGBoost to enhance the robustness of Machine Reading Comprehension models against adversarial attacks and data shifts, improving performance across multiple datasets.

Contribution

It proposes a novel calibrator method that combines manual and learned features for reranking, improving robustness without degrading original dataset performance.

Findings

Performance improved by over 10% on adversarial datasets

Enhanced robustness without loss on original data

Effective reranking method for MRC models

Abstract

Machine Reading Comprehension(MRC) has achieved a remarkable result since some powerful models, such as BERT, are proposed. However, these models are not robust enough and vulnerable to adversarial input perturbation and generalization examples. Some works tried to improve the performance on specific types of data by adding some related examples into training data while it leads to degradation on the original dataset, because the shift of data distribution makes the answer ranking based on the softmax probability of model unreliable. In this paper, we propose a method to improve the robustness by using a calibrator as the post-hoc reranker, which is implemented based on XGBoost model. The calibrator combines both manual features and representation learning features to rerank candidate results. Experimental results on adversarial datasets show that our model can achieve performance improvement by more than 10\% and also make improvement on the original and generalization datasets.