Towards Human-Level Text Coding with LLMs: The Case of Fatherhood Roles in Public Policy Documents

TL;DR

This paper demonstrates that with proper prompting, large language models can effectively perform complex text coding tasks in political science, matching or surpassing human performance while being faster and more scalable.

Contribution

It introduces a practical workflow for optimizing LLM prompts for political text coding and compares GPT and open-source models, enabling scalable annotation.

Findings

LLMs can match or outperform human coders with detailed prompts.

Prompting with a comprehensive codebook improves coding accuracy.

Open-source LLMs offer a trade-off between performance and accessibility.

Abstract

Recent advances in large language models (LLMs) like GPT-3.5 and GPT-4 promise automation with better results and less programming, opening up new opportunities for text analysis in political science. In this study, we evaluate LLMs on three original coding tasks involving typical complexities encountered in political science settings: a non-English language, legal and political jargon, and complex labels based on abstract constructs. Along the paper, we propose a practical workflow to optimize the choice of the model and the prompt. We find that the best prompting strategy consists of providing the LLMs with a detailed codebook, as the one provided to human coders. In this setting, an LLM can be as good as or possibly better than a human annotator while being much faster, considerably cheaper, and much easier to scale to large amounts of text. We also provide a comparison of GPT and popular open-source LLMs, discussing the trade-offs in the model's choice. Our software allows LLMs to be easily used as annotators and is publicly available: https://github.com/lorelupo/pappa.