Comparative Insights from 12 Machine Learning Models in Extracting Economic Ideology from Political Text

TL;DR

This paper systematically compares 12 machine learning models, including generative, fine-tuned, and zero-shot, in extracting economic ideology from UK political manifestos, highlighting their strengths, limitations, and best practices.

Contribution

It provides a comprehensive evaluation of various models for political text analysis, revealing the superior performance of generative models and the trade-offs of fine-tuning and zero-shot approaches.

Findings

Generative models like GPT-4o outperform others in accuracy.

Fine-tuning improves performance but depends on training data size.

Zero-shot models struggle with domain-specific signals.

Abstract

This study conducts a systematic assessment of the capabilities of 12 machine learning models and model variations in detecting economic ideology. As an evaluation benchmark, I use manifesto data spanning six elections in the United Kingdom and pre-annotated by expert and crowd coders. The analysis assesses the performance of several generative, fine-tuned, and zero-shot models at the granular and aggregate levels. The results show that generative models such as GPT-4o and Gemini 1.5 Flash consistently outperform other models against all benchmarks. However, they pose issues of accessibility and resource availability. Fine-tuning yielded competitive performance and offers a reliable alternative through domain-specific optimization. But its dependency on training data severely limits scalability. Zero-shot models consistently face difficulties with identifying signals of economic ideology, often resulting in negative associations with human coding. Using general knowledge for the domain-specific task of ideology scaling proved to be unreliable. Other key findings include considerable within-party variation, fine-tuning benefiting from larger training data, and zero-shot's sensitivity to prompt content. The assessments include the strengths and limitations of each model and derive best-practices for automated analyses of political content.