Can LLMs Understand the Impact of Trauma? Costs and Benefits of LLMs Coding the Interviews of Firearm Violence Survivors

TL;DR

This study evaluates the use of open-source large language models for coding interviews with firearm violence survivors, revealing potential but significant limitations in relevance and ethical considerations.

Contribution

It provides an empirical assessment of LLMs in qualitative coding of sensitive interviews, highlighting both capabilities and ethical challenges.

Findings

LLMs can identify some important codes but with low overall relevance.

Relevance of coding is highly sensitive to data processing choices.

Guardrails in LLMs cause substantial narrative erasure.

Abstract

Firearm violence is a pressing public health issue, yet research into survivors' lived experiences remains underfunded and difficult to scale. Qualitative research, including in-depth interviews, is a valuable tool for understanding the personal and societal consequences of community firearm violence and designing effective interventions. However, manually analyzing these narratives through thematic analysis and inductive coding is time-consuming and labor-intensive. Recent advancements in large language models (LLMs) have opened the door to automating this process, though concerns remain about whether these models can accurately and ethically capture the experiences of vulnerable populations. In this study, we assess the use of open-source LLMs to inductively code interviews with 21 Black men who have survived community firearm violence. Our results demonstrate that while some configurations of LLMs can identify important codes, overall relevance remains low and is highly sensitive to data processing. Furthermore, LLM guardrails lead to substantial narrative erasure. These findings highlight both the potential and limitations of LLM-assisted qualitative coding and underscore the ethical challenges of applying AI in research involving marginalized communities.