ANNA: Enhanced Language Representation for Question Answering

TL;DR

This paper introduces ANNA, an enhanced pre-trained language model with a neighbor-aware mechanism and extended pre-training tasks, achieving state-of-the-art results on question answering benchmarks like SQuAD 1.1 and 2.0.

Contribution

The paper presents a novel neighbor-aware mechanism and an extended pre-training task that, when combined, improve language model performance on question answering tasks.

Findings

Achieved 95.7% F1 and 90.6% EM on SQuAD 1.1

Outperformed models like RoBERTa, ALBERT, ELECTRA, XLNet on SQuAD 2.0

Demonstrated the effectiveness of joint pre-training approaches

Abstract

Pre-trained language models have brought significant improvements in performance in a variety of natural language processing tasks. Most existing models performing state-of-the-art results have shown their approaches in the separate perspectives of data processing, pre-training tasks, neural network modeling, or fine-tuning. In this paper, we demonstrate how the approaches affect performance individually, and that the language model performs the best results on a specific question answering task when those approaches are jointly considered in pre-training models. In particular, we propose an extended pre-training task, and a new neighbor-aware mechanism that attends neighboring tokens more to capture the richness of context for pre-training language modeling. Our best model achieves new state-of-the-art results of 95.7\% F1 and 90.6\% EM on SQuAD 1.1 and also outperforms existing pre-trained language models such as RoBERTa, ALBERT, ELECTRA, and XLNet on the SQuAD 2.0 benchmark.