Look to the Right: Mitigating Relative Position Bias in Extractive Question Answering

TL;DR

This paper identifies relative position bias as a spurious correlation in extractive QA models and proposes an ensemble-based debiasing method to improve their generalization to unseen relative positions.

Contribution

It introduces a novel bias in QA models related to answer relative positions and presents a debiasing approach that does not require prior distribution knowledge.

Findings

The relative position bias significantly affects model performance on unseen positions.

The proposed ensemble-based method effectively reduces reliance on relative position bias.

Results show improved generalization on biased and full SQuAD datasets.

Abstract

Extractive question answering (QA) models tend to exploit spurious correlations to make predictions when a training set has unintended biases. This tendency results in models not being generalizable to examples where the correlations do not hold. Determining the spurious correlations QA models can exploit is crucial in building generalizable QA models in real-world applications; moreover, a method needs to be developed that prevents these models from learning the spurious correlations even when a training set is biased. In this study, we discovered that the relative position of an answer, which is defined as the relative distance from an answer span to the closest question-context overlap word, can be exploited by QA models as superficial cues for making predictions. Specifically, we find that when the relative positions in a training set are biased, the performance on examples with relative positions unseen during training is significantly degraded. To mitigate the performance degradation for unseen relative positions, we propose an ensemble-based debiasing method that does not require prior knowledge about the distribution of relative positions. We demonstrate that the proposed method mitigates the models' reliance on relative positions using the biased and full SQuAD dataset. We hope that this study can help enhance the generalization ability of QA models in real-world applications.