XRJL-HKUST at SemEval-2021 Task 4: WordNet-Enhanced Dual Multi-head Co-Attention for Reading Comprehension of Abstract Meaning

TL;DR

This paper introduces WN-DUMA, a reading comprehension model that enhances passage-question understanding with dual multi-head co-attention and WordNet-based abstract concept knowledge, achieving high accuracy on SemEval-2021 tasks.

Contribution

The paper proposes a novel stacked attention mechanism and integrates WordNet definitions to improve comprehension of abstract meanings in reading tasks.

Findings

Achieved 86.67% accuracy on subtask 1

Achieved 89.99% accuracy on subtask 2

Enhanced understanding of abstract concepts through external knowledge

Abstract

This paper presents our submitted system to SemEval 2021 Task 4: Reading Comprehension of Abstract Meaning. Our system uses a large pre-trained language model as the encoder and an additional dual multi-head co-attention layer to strengthen the relationship between passages and question-answer pairs, following the current state-of-the-art model DUMA. The main difference is that we stack the passage-question and question-passage attention modules instead of calculating parallelly to simulate re-considering process. We also add a layer normalization module to improve the performance of our model. Furthermore, to incorporate our known knowledge about abstract concepts, we retrieve the definitions of candidate answers from WordNet and feed them to the model as extra inputs. Our system, called WordNet-enhanced DUal Multi-head Co-Attention (WN-DUMA), achieves 86.67% and 89.99% accuracy on the official blind test set of subtask 1 and subtask 2 respectively.