ApplicaAI at SemEval-2020 Task 11: On RoBERTa-CRF, Span CLS and Whether Self-Training Helps Them

TL;DR

This paper describes winning and high-ranking systems for propaganda technique classification and span identification tasks, utilizing innovative RoBERTa-CRF architecture, Span CLS layers, and semi-supervised self-training to improve performance.

Contribution

It introduces the novel use of RoBERTa-CRF and Span CLS layers in propaganda detection, demonstrating their effectiveness through competitive results.

Findings

RoBERTa-CRF architecture enhances span identification accuracy

Span CLS layers improve propaganda technique classification

Self-training boosts model performance with limited labeled data

Abstract

This paper presents the winning system for the propaganda Technique Classification (TC) task and the second-placed system for the propaganda Span Identification (SI) task. The purpose of TC task was to identify an applied propaganda technique given propaganda text fragment. The goal of SI task was to find specific text fragments which contain at least one propaganda technique. Both of the developed solutions used semi-supervised learning technique of self-training. Interestingly, although CRF is barely used with transformer-based language models, the SI task was approached with RoBERTa-CRF architecture. An ensemble of RoBERTa-based models was proposed for the TC task, with one of them making use of Span CLS layers we introduce in the present paper. In addition to describing the submitted systems, an impact of architectural decisions and training schemes is investigated along with remarks regarding training models of the same or better quality with lower computational budget. Finally, the results of error analysis are presented.