IIE-NLP-Eyas at SemEval-2021 Task 4: Enhancing PLM for ReCAM with Special Tokens, Re-Ranking, Siamese Encoders and Back Translation

TL;DR

This paper enhances pre-trained language models for reading comprehension of abstract concepts by using special tokens, re-ranking, Siamese encoders, and back translation, leading to improved performance on SemEval-2021 tasks.

Contribution

It introduces novel adaptation techniques for PLMs, including special tokens and multiple finetuning tricks, tailored for understanding abstract concepts in reading comprehension tasks.

Findings

Achieved top-10 rankings in SemEval-2021 Task 4.

Significant performance improvements over baseline models.

Effective use of special tokens and re-ranking methods.

Abstract

This paper introduces our systems for all three subtasks of SemEval-2021 Task 4: Reading Comprehension of Abstract Meaning. To help our model better represent and understand abstract concepts in natural language, we well-design many simple and effective approaches adapted to the backbone model (RoBERTa). Specifically, we formalize the subtasks into the multiple-choice question answering format and add special tokens to abstract concepts, then, the final prediction of question answering is considered as the result of subtasks. Additionally, we employ many finetuning tricks to improve the performance. Experimental results show that our approaches achieve significant performance compared with the baseline systems. Our approaches achieve eighth rank on subtask-1 and tenth rank on subtask-2.