Identifying Fine-grained Forms of Populism in Political Discourse: A Case Study on Donald Trump's Presidential Campaigns

TL;DR

This study evaluates large language models' ability to identify and classify nuanced populist discourse in political speeches, revealing their limitations and the effectiveness of fine-tuned classifiers, with implications for social science research.

Contribution

It introduces novel datasets for populist discourse and compares various models, demonstrating the superiority of fine-tuned classifiers over instruction-tuned LLMs in this task.

Findings

Fine-tuned RoBERTa outperforms instruction-tuned LLMs in populist classification.

Instruction-tuned LLMs show greater robustness on out-of-domain political speeches.

Models reveal insights into Donald Trump's strategic use of populist rhetoric.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of instruction-following tasks, yet their grasp of nuanced social science concepts remains underexplored. This paper examines whether LLMs can identify and classify fine-grained forms of populism, a complex and contested concept in both academic and media debates. To this end, we curate and release novel datasets specifically designed to capture populist discourse. We evaluate a range of pre-trained (large) language models, both open-weight and proprietary, across multiple prompting paradigms. Our analysis reveals notable variation in performance, highlighting the limitations of LLMs in detecting populist discourse. We find that a fine-tuned RoBERTa classifier vastly outperforms all new-era instruction-tuned LLMs, unless fine-tuned. Additionally, we apply our best-performing model to analyze campaign speeches by Donald Trump, extracting valuable insights into his strategic use of populist rhetoric. Finally, we assess the generalizability of these models by benchmarking them on campaign speeches by European politicians, offering a lens into cross-context transferability in political discourse analysis. In this setting, we find that instruction-tuned LLMs exhibit greater robustness on out-of-domain data.