NLP-IIS@UT at SemEval-2021 Task 4: Machine Reading Comprehension using the Long Document Transformer

TL;DR

This paper describes the use of the Longformer model for machine reading comprehension on lengthy documents, significantly improving accuracy over baseline models in SemEval-2021 Task 4.

Contribution

We applied the Longformer model to handle long contexts in reading comprehension, demonstrating substantial accuracy improvements over traditional models like BERT.

Findings

Accuracy increased from around 23% to over 70% on subtask 1.

Accuracy improved from around 23% to over 64% on subtask 2.

Longformer effectively processes long documents for comprehension tasks.

Abstract

This paper presents a technical report of our submission to the 4th task of SemEval-2021, titled: Reading Comprehension of Abstract Meaning. In this task, we want to predict the correct answer based on a question given a context. Usually, contexts are very lengthy and require a large receptive field from the model. Thus, common contextualized language models like BERT miss fine representation and performance due to the limited capacity of the input tokens. To tackle this problem, we used the Longformer model to better process the sequences. Furthermore, we utilized the method proposed in the Longformer benchmark on Wikihop dataset which improved the accuracy on our task data from 23.01% and 22.95% achieved by the baselines for subtask 1 and 2, respectively, to 70.30% and 64.38%.