BUT-FIT at SemEval-2020 Task 4: Multilingual commonsense

TL;DR

This paper presents the BUT-FIT team's approach to SemEval-2020 Task 4, utilizing pretrained language models, data augmentation, and multilingual techniques, achieving top BLEU scores and analyzing evaluation metrics and errors.

Contribution

The paper introduces a multilingual approach using ALBERT and machine translation, and proposes new evaluation metrics and reranking methods for commonsense validation and explanation.

Findings

Multilingual models with machine translation perform well with minimal accuracy loss.

BART-based system achieved 1st place in BLEU score ranking.

Proposed new evaluation metric correlates better with human judgment.

Abstract

This paper describes work of the BUT-FIT's team at SemEval 2020 Task 4 - Commonsense Validation and Explanation. We participated in all three subtasks. In subtasks A and B, our submissions are based on pretrained language representation models (namely ALBERT) and data augmentation. We experimented with solving the task for another language, Czech, by means of multilingual models and machine translated dataset, or translated model inputs. We show that with a strong machine translation system, our system can be used in another language with a small accuracy loss. In subtask C, our submission, which is based on pretrained sequence-to-sequence model (BART), ranked 1st in BLEU score ranking, however, we show that the correlation between BLEU and human evaluation, in which our submission ended up 4th, is low. We analyse the metrics used in the evaluation and we propose an additional score based on model from subtask B, which correlates well with our manual ranking, as well as reranking method based on the same principle. We performed an error and dataset analysis for all subtasks and we present our findings.