Automated Thematic Analyses Using LLMs: Xylazine Wound Management Social Media Chatter Use Case

TL;DR

This study demonstrates that large language models, especially GPT-4o with few-shot prompting, can effectively automate thematic analysis of social media data, closely matching expert classifications and offering a scalable qualitative research tool.

Contribution

It introduces a novel approach to using LLMs for inductive thematic analysis, validating their performance against expert coding in social media datasets.

Findings

GPT-4o achieved 90.9% accuracy in thematic classification.

Model distributions closely matched expert classifications for key themes.

Few-shot prompting significantly improved model performance.

Abstract

Background Large language models (LLMs) face challenges in inductive thematic analysis, a task requiring deep interpretive and domain-specific expertise. We evaluated the feasibility of using LLMs to replicate expert-driven thematic analysis of social media data. Methods Using two temporally non-intersecting Reddit datasets on xylazine (n=286 and n=686, for model optimization and validation, respectively) with twelve expert-derived themes, we evaluated five LLMs against expert coding. We modeled the task as a series of binary classifications, rather than a single, multi-label classification, employing zero-, single-, and few-shot prompting strategies and measuring performance via accuracy, precision, recall, and F1-score. Results On the validation set, GPT-4o with two-shot prompting performed best (accuracy: 90.9%; F1-score: 0.71). For high-prevalence themes, model-derived thematic distributions closely mirrored expert classifications (e.g., xylazine use: 13.6% vs. 17.8%; MOUD use: 16.5% vs. 17.8%). Conclusions Our findings suggest that few-shot LLM-based approaches can automate thematic analyses, offering a scalable supplement for qualitative research. Keywords: thematic analysis, large language models, natural language processing, qualitative analysis, social media, prompt engineering, public health