A Human-Centered Workflow for Using Large Language Models in Content Analysis

TL;DR

This paper introduces a human-centered workflow for applying large language models (LLMs) via APIs in content analysis, emphasizing rigorous supervision, validation, and transparency across tasks like annotation, summarization, and information extraction.

Contribution

It presents a comprehensive, human-centered methodology for using LLMs in content analysis, including validation procedures, best practices, and practical tools for researchers.

Findings

Workflow enhances rigor and transparency in LLM-based content analysis

Validation procedures address LLM limitations like hallucination and prompt sensitivity

Practical tools include prompt library and Python code for implementation

Abstract

While many researchers use Large Language Models (LLMs) through chat-based access, their real potential lies in leveraging LLMs via application programming interfaces (APIs). This paper conceptualizes LLMs as universal text processing machines and presents a comprehensive workflow for employing LLMs in three qualitative and quantitative content analysis tasks: (1) annotation (an umbrella term for qualitative coding, labeling and text classification), (2) summarization, and (3) information extraction. The workflow is explicitly human-centered. Researchers design, supervise, and validate each stage of the LLM process to ensure rigor and transparency. Our approach synthesizes insights from extensive methodological literature across multiple disciplines: political science, sociology, computer science, psychology, and management. We outline validation procedures and best practices to address key limitations of LLMs, such as their black-box nature, prompt sensitivity, and tendency to hallucinate. To facilitate practical implementation, we provide supplementary materials, including a prompt library and Python code in Jupyter Notebook format, accompanied by detailed usage instructions.