Are Large Language Models Good at Detecting Propaganda?

TL;DR

This paper evaluates the effectiveness of large language models in detecting propaganda techniques in news articles, revealing limited overall performance but some strengths in specific techniques compared to baseline models.

Contribution

It provides a comparative analysis of LLMs versus traditional models for propaganda detection, highlighting their strengths and limitations in this task.

Findings

GPT-4 has higher F1 scores than GPT-3.5 and Claude 3 but underperforms compared to the RoBERTa-CRF baseline.

All three LLMs outperform the MGN baseline in detecting 'name-calling' propaganda.

GPT-3.5 and GPT-4 outperform the MGN baseline in detecting 'appeal to fear' and 'flag-waving' techniques.

Abstract

Propagandists use rhetorical devices that rely on logical fallacies and emotional appeals to advance their agendas. Recognizing these techniques is key to making informed decisions. Recent advances in Natural Language Processing (NLP) have enabled the development of systems capable of detecting manipulative content. In this study, we look at several Large Language Models and their performance in detecting propaganda techniques in news articles. We compare the performance of these LLMs with transformer-based models. We find that, while GPT-4 demonstrates superior F1 scores (F1=0.16) compared to GPT-3.5 and Claude 3 Opus, it does not outperform a RoBERTa-CRF baseline (F1=0.67). Additionally, we find that all three LLMs outperform a MultiGranularity Network (MGN) baseline in detecting instances of one out of six propaganda techniques (name-calling), with GPT-3.5 and GPT-4 also outperforming the MGN baseline in detecting instances of appeal to fear and flag-waving.