A Methodological Guide on Using Large Language Models for Text Annotation in the Social Sciences and Humanities with Python and R

TL;DR

This paper offers a detailed, practical guide with code examples for social science and humanities researchers to effectively use large language models for text annotation, addressing challenges and ensuring reliable analysis.

Contribution

It provides a comprehensive, step-by-step methodology with Python and R code for using LLMs in text annotation, including quality evaluation and integration into statistical analysis.

Findings

Guidance on designing prompts and evaluating annotation quality

Strategies for integrating LLM annotations into statistical models

Best practices for scaling and managing costs in LLM-based annotation

Abstract

Large language models (LLMs) have become an essential tool for social science and humanities (SSH) researchers who work with text. One particularly valuable application is automating text annotation, a traditionally time-consuming step in preparing data for empirical analysis. Yet many SSH researchers face two challenges: getting started with LLMs and understanding how to address their limitations. Practically, the rapid pace of model development can make LLMs seem inaccessible or intimidating, while even experienced users may overlook how annotation errors can bias downstream statistical analyses (e.g., regression estimates and $p$-values), even when annotation accuracy appears high. This paper provides a comprehensive, step-by-step methodological guide for using LLMs for text annotation in SSH research, with clear Python and R code snippets. We cover (1) how LLMs work and what they can and cannot do; (2) how to identify an LLM-suitable research project and establish minimum data and computational requirements; (3) how to design prompts and run annotation tasks; (4) how to evaluate annotation quality and iteratively refine prompts without overfitting; (5) how to integrate LLM annotations into downstream statistical analyses while accounting for annotation error; and (6) how to manage cost, efficiency, and reproducibility when scaling up annotation. Throughout, we provide intuitive methodological reasoning, concrete examples, code snippets, and best-practice guidance to help researchers confidently and transparently incorporate LLM-based annotation into their scientific workflows.