When to Fold'em: How to answer Unanswerable questions

TL;DR

This paper compares three question-answering models trained on SQuAD2.0, introduces a novel fine-tuning approach that improves F1 scores by 2% with less training, and highlights the effectiveness of re-initializing specific model layers.

Contribution

A new fine-tuning method involving re-initializing select layers of a shared language model, leading to improved performance and reduced training time.

Findings

Achieved a 2% increase in SQuAD2.0 F1 score.

Demonstrated the effectiveness of re-initializing layers in pre-trained models.

Compared BIDAF, DocumentQA, and ALBERT Retro-Reader models.

Abstract

We present 3 different question-answering models trained on the SQuAD2.0 dataset -- BIDAF, DocumentQA and ALBERT Retro-Reader -- demonstrating the improvement of language models in the past three years. Through our research in fine-tuning pre-trained models for question-answering, we developed a novel approach capable of achieving a 2% point improvement in SQuAD2.0 F1 in reduced training time. Our method of re-initializing select layers of a parameter-shared language model is simple yet empirically powerful.