Assessing the Impact of Conspiracy Theories Using Large Language Models

TL;DR

This paper explores how large language models can be used to evaluate the impact of conspiracy theories by developing datasets, designing assessment strategies, and analyzing biases and accuracy in impact predictions.

Contribution

It introduces datasets of conspiracy theories with impact annotations and proposes tailored LLM-based strategies for impact assessment, highlighting their strengths and biases.

Findings

Multi-step reasoning improves impact assessment accuracy.

LLMs tend to bias impact scores based on prompt order.

Emotionally charged and verbose CTs are less accurately assessed.

Abstract

Measuring the relative impact of CTs is important for prioritizing responses and allocating resources effectively, especially during crises. However, assessing the actual impact of CTs on the public poses unique challenges. It requires not only the collection of CT-specific knowledge but also diverse information from social, psychological, and cultural dimensions. Recent advancements in large language models (LLMs) suggest their potential utility in this context, not only due to their extensive knowledge from large training corpora but also because they can be harnessed for complex reasoning. In this work, we develop datasets of popular CTs with human-annotated impacts. Borrowing insights from human impact assessment processes, we then design tailored strategies to leverage LLMs for performing human-like CT impact assessments. Through rigorous experiments, we textit{discover that an impact assessment mode using multi-step reasoning to analyze more CT-related evidence critically produces accurate results; and most LLMs demonstrate strong bias, such as assigning higher impacts to CTs presented earlier in the prompt, while generating less accurate impact assessments for emotionally charged and verbose CTs.