Computational frame analysis revisited: On LLMs for studying news coverage

TL;DR

This study evaluates the effectiveness of large language models in media frame analysis, comparing them to traditional methods and manual coding, and offers guidance on their appropriate use in research workflows.

Contribution

It provides a systematic comparison of LLMs with traditional and manual methods for media frame analysis using a novel dataset from the US Mpox epidemic coverage.

Findings

LLMs are outperformed by manual coders and smaller models in frame analysis.

Human validation remains essential for model selection.

Different approaches are suitable for different analytical tasks.

Abstract

Computational approaches have previously shown various promises and pitfalls when it comes to the reliable identification of media frames. Generative LLMs like GPT and Claude are increasingly being used as content analytical tools, but how effective are they for frame analysis? We address this question by systematically evaluating them against their computational predecessors: bag-of-words models and encoder-only transformers; and traditional manual coding procedures. Our analysis rests on a novel gold standard dataset that we inductively and iteratively developed through the study, investigating six months of news coverage of the US Mpox epidemic of 2022. While we discover some potential applications for generative LLMs, we demonstrate that they were consistently outperformed by manual coders, and in some instances, by smaller language models. Some form of human validation was always necessary to determine appropriate model choice. Additionally, by examining how the suitability of various approaches depended on the nature of different tasks that were part of our frame analytical workflow, we provide insights as to how researchers may leverage the complementarity of these approaches to use them in tandem. We conclude by endorsing a methodologically pluralistic approach and put forth a roadmap for computational frame analysis for researchers going forward.