QiaoNing at SemEval-2020 Task 4: Commonsense Validation and Explanation system based on ensemble of language model

TL;DR

This paper presents an ensemble of pretrained language models for the SemEval-2020 Task 4, achieving high accuracy in commonsense validation and explanation tasks by fine-tuning BERT, XLNet, RoBERTa, and ALBERT.

Contribution

The paper introduces an ensemble approach using multiple pretrained language models for commonsense validation and explanation, with detailed comparison and fine-tuning strategies.

Findings

Ensembled model achieved 95.9% accuracy on subtask A.

Fine-tuning pretrained models improves performance on commonsense tasks.

Ensemble approach outperforms individual models in accuracy.

Abstract

In this paper, we present language model system submitted to SemEval-2020 Task 4 competition: "Commonsense Validation and Explanation". We participate in two subtasks for subtask A: validation and subtask B: Explanation. We implemented with transfer learning using pretrained language models (BERT, XLNet, RoBERTa, and ALBERT) and fine-tune them on this task. Then we compared their characteristics in this task to help future researchers understand and use these models more properly. The ensembled model better solves this problem, making the model's accuracy reached 95.9% on subtask A, which just worse than human's by only 3% accuracy.