XD at SemEval-2020 Task 12: Ensemble Approach to Offensive Language Identification in Social Media Using Transformer Encoders

TL;DR

This paper develops an ensemble of transformer-based models for offensive language detection in social media, achieving high accuracy and outperforming many systems in a shared task.

Contribution

It introduces a novel ensemble approach combining multiple transformer encoders for improved offensive language identification.

Findings

Ensemble model outperforms individual models by up to 8.6% on development set.

Achieves macro-F1 score of 90.9% on the test set.

High-performing system among 85 participants in shared task A.

Abstract

This paper presents six document classification models using the latest transformer encoders and a high-performing ensemble model for a task of offensive language identification in social media. For the individual models, deep transformer layers are applied to perform multi-head attentions. For the ensemble model, the utterance representations taken from those individual models are concatenated and fed into a linear decoder to make the final decisions. Our ensemble model outperforms the individual models and shows up to 8.6% improvement over the individual models on the development set. On the test set, it achieves macro-F1 of 90.9% and becomes one of the high performing systems among 85 participants in the sub-task A of this shared task. Our analysis shows that although the ensemble model significantly improves the accuracy on the development set, the improvement is not as evident on the test set.