Understanding BERT performance in propaganda analysis

TL;DR

This paper evaluates a fine-tuned BERT model for sentence-level propaganda analysis, achieving competitive results and providing insights into its strengths and limitations in distinguishing propaganda from opinion and quotations.

Contribution

It introduces a BERT-based system for propaganda detection, analyzes its performance, and investigates its errors to understand its capabilities and limitations.

Findings

BERT model scored 0.62 F1, ranking third in shared task.

System tends to misclassify opinion pieces as propaganda.

Difficulty in distinguishing quotations of propaganda from actual propaganda techniques.

Abstract

In this paper, we describe our system used in the shared task for fine-grained propaganda analysis at sentence level. Despite the challenging nature of the task, our pretrained BERT model (team YMJA) fine tuned on the training dataset provided by the shared task scored 0.62 F1 on the test set and ranked third among 25 teams who participated in the contest. We present a set of illustrative experiments to better understand the performance of our BERT model on this shared task. Further, we explore beyond the given dataset for false-positive cases that likely to be produced by our system. We show that despite the high performance on the given testset, our system may have the tendency of classifying opinion pieces as propaganda and cannot distinguish quotations of propaganda speech from actual usage of propaganda techniques.