Magic Words or Methodical Work? Challenging Conventional Wisdom in LLM-Based Political Text Annotation

TL;DR

This study systematically evaluates how different implementation choices affect LLM-based political text annotation, revealing that interaction effects are more influential than individual factors, and proposing a validation framework for researchers.

Contribution

It provides a controlled comparison of models, prompts, and learning approaches, highlighting the importance of interaction effects and introducing a validation-first framework.

Findings

Interaction effects dominate main effects in pipeline choices.

No single model or prompt style is best across all tasks.

Larger models are not always more resource-efficient or better performing.

Abstract

Political scientists are rapidly adopting large language models (LLMs) for text annotation, yet the sensitivity of annotation results to implementation choices remains poorly understood. Most evaluations test a single model or configuration; how model choice, model size, learning approach, and prompt style interact, and whether popular "best practices" survive controlled comparison, are largely unexplored. We present a controlled evaluation of these pipeline choices, testing six open-weight models across four political science annotation tasks under identical quantisation, hardware, and prompt-template conditions. Our central finding is methodological: interaction effects dominate main effects, so seemingly reasonable pipeline choices can become consequential researcher degrees of freedom. No single model, prompt style, or learning approach is uniformly superior, and the best-performing model varies across tasks. Two corollaries follow. First, model size is an unreliable guide both to cost and to performance: cross-family efficiency differences are so large that some larger models are less resource-intensive than much smaller alternatives, while within model families mid-range variants often match or exceed larger counterparts. Second, widely recommended prompt engineering techniques yield inconsistent and sometimes negative effects on annotation performance. We use these benchmark results to develop a validation-first framework - with a principled ordering of pipeline decisions, guidance on prompt freezing and held-out evaluation, reporting standards, and open-source tools - to help researchers navigate this decision space transparently.