Large Language Models Reveal Information Operation Goals, Tactics, and Narrative Frames

TL;DR

This paper investigates the use of GPT-3.5 to analyze and interpret information operations on social media, aiming to automate the extraction of campaign goals, tactics, and narratives to better understand influence campaigns.

Contribution

It demonstrates that large language models can effectively analyze complex information operations, providing a more objective and comprehensive approach than manual analysis.

Findings

GPT-3.5 shows high agreement with ground truth in campaign annotation.

LLMs can extract goals, tactics, and narratives from multilingual social media data.

Automated analysis can reveal changes in campaign strategies over time.

Abstract

Adversarial information operations can destabilize societies by undermining fair elections, manipulating public opinions on policies, and promoting scams. Despite their widespread occurrence and potential impacts, our understanding of influence campaigns is limited by manual analysis of messages and subjective interpretation of their observable behavior. In this paper, we explore whether these limitations can be mitigated with large language models (LLMs), using GPT-3.5 as a case-study for coordinated campaign annotation. We first use GPT-3.5 to scrutinize 126 identified information operations spanning over a decade. We utilize a number of metrics to quantify the close (if imperfect) agreement between LLM and ground truth descriptions. We next extract coordinated campaigns from two large multilingual datasets from X (formerly Twitter) that respectively discuss the 2022 French election and 2023 Balikaran Philippine-U.S. military exercise in 2023. For each coordinated campaign, we use GPT-3.5 to analyze posts related to a specific concern and extract goals, tactics, and narrative frames, both before and after critical events (such as the date of an election). While the GPT-3.5 sometimes disagrees with subjective interpretation, its ability to summarize and interpret demonstrates LLMs' potential to extract higher-order indicators from text to provide a more complete picture of the information campaigns compared to previous methods.