Generative Large Language Models (gLLMs) in Content Analysis: A Practical Guide for Communication Research

TL;DR

This paper explores how generative large language models like ChatGPT are transforming content analysis in communication research by offering faster, cost-effective, and more nuanced coding capabilities, while also addressing methodological challenges.

Contribution

It provides a comprehensive guide on integrating gLLMs into content analysis, highlighting best practices and addressing key challenges to improve research validity and reliability.

Findings

gLLMs outperform traditional coding methods in speed and cost

gLLMs can decode implicit meanings and contextual information

The paper offers a best-practice framework for using gLLMs in research

Abstract

Generative Large Language Models (gLLMs), such as ChatGPT, are increasingly being used in communication research for content analysis. Studies show that gLLMs can outperform both crowd workers and trained coders, such as research assistants, on various coding tasks relevant to communication science, often at a fraction of the time and cost. Additionally, gLLMs can decode implicit meanings and contextual information, be instructed using natural language, deployed with only basic programming skills, and require little to no annotated data beyond a validation dataset - constituting a paradigm shift in automated content analysis. Despite their potential, the integration of gLLMs into the methodological toolkit of communication research remains underdeveloped. In gLLM-assisted quantitative content analysis, researchers must address at least seven critical challenges that impact result quality: (1) codebook development, (2) prompt engineering, (3) model selection, (4) parameter tuning, (5) iterative refinement, (6) validation of the model's reliability, and optionally, (7) performance enhancement. This paper synthesizes emerging research on gLLM-assisted quantitative content analysis and proposes a comprehensive best-practice guide to navigate these challenges. Our goal is to make gLLM-based content analysis more accessible to a broader range of communication researchers and ensure adherence to established disciplinary quality standards of validity, reliability, reproducibility, and research ethics.