Classifying Conspiratorial Narratives At Scale: False Alarms and Erroneous Connections

TL;DR

This paper develops a large-scale classification method for conspiracy theory discussions on Reddit, comparing BERT-based models and GPT, revealing strengths and flaws in detecting nuanced conspiratorial content.

Contribution

It introduces a general scheme for classifying conspiracy-related discussions based on authors' perspectives and compares traditional models with GPT in this context.

Findings

Only one-third of conspiracy-related posts are classified as positive.

BERT-based classifiers perform comparably to GPT in detecting conspiratorial content.

GPT shows significant flaws in logical reasoning despite its contextual strengths.

Abstract

Online discussions frequently involve conspiracy theories, which can contribute to the proliferation of belief in them. However, not all discussions surrounding conspiracy theories promote them, as some are intended to debunk them. Existing research has relied on simple proxies or focused on a constrained set of signals to identify conspiracy theories, which limits our understanding of conspiratorial discussions across different topics and online communities. This work establishes a general scheme for classifying discussions related to conspiracy theories based on authors' perspectives on the conspiracy belief, which can be expressed explicitly through narrative elements, such as the agent, action, or objective, or implicitly through references to known theories, such as chemtrails or the New World Order. We leverage human-labeled ground truth to train a BERT-based model for classifying online CTs, which we then compared to the Generative Pre-trained Transformer machine (GPT) for detecting online conspiratorial content. Despite GPT's known strengths in its expressiveness and contextual understanding, our study revealed significant flaws in its logical reasoning, while also demonstrating comparable strengths from our classifiers. We present the first large-scale classification study using posts from the most active conspiracy-related Reddit forums and find that only one-third of the posts are classified as positive. This research sheds light on the potential applications of large language models in tasks demanding nuanced contextual comprehension.