Ensemble ALBERT on SQuAD 2.0

TL;DR

This paper enhances question answering performance on SQuAD 2.0 by fine-tuning ALBERT models with additional layers and ensemble methods, achieving state-of-the-art results.

Contribution

It introduces multiple layered models based on ALBERT and applies ensemble algorithms to significantly improve SQuAD 2.0 performance.

Findings

Best model achieved F1 score of 88.435 on dev set

Ensemble methods improved F1 to 90.123 on leaderboard

Model variations outperform baseline ALBERT models

Abstract

Machine question answering is an essential yet challenging task in natural language processing. Recently, Pre-trained Contextual Embeddings (PCE) models like Bidirectional Encoder Representations from Transformers (BERT) and A Lite BERT (ALBERT) have attracted lots of attention due to their great performance in a wide range of NLP tasks. In our Paper, we utilized the fine-tuned ALBERT models and implemented combinations of additional layers (e.g. attention layer, RNN layer) on top of them to improve model performance on Stanford Question Answering Dataset (SQuAD 2.0). We implemented four different models with different layers on top of ALBERT-base model, and two other models based on ALBERT-xlarge and ALBERT-xxlarge. We compared their performance to our baseline model ALBERT-base-v2 + ALBERT-SQuAD-out with details. Our best-performing individual model is ALBERT-xxlarge + ALBERT-SQuAD-out, which achieved an F1 score of 88.435 on the dev set. Furthermore, we have implemented three different ensemble algorithms to boost overall performance. By passing in several best-performing models' results into our weighted voting ensemble algorithm, our final result ranks first on the Stanford CS224N Test PCE SQuAD Leaderboard with F1 = 90.123.