Using Instruction-Tuned Large Language Models to Identify Indicators of Vulnerability in Police Incident Narratives

TL;DR

This study evaluates instruction-tuned large language models for classifying police incident narratives, showing they can effectively support human coding, reduce resource use, and exhibit limited bias based on race and gender.

Contribution

The paper demonstrates the effectiveness of IT-LLMs in supporting qualitative coding of police narratives and assesses their bias, offering a scalable, transparent approach for large unstructured datasets.

Findings

IT-LLMs effectively support human coding of police narratives.

They are highly accurate at identifying narratives without vulnerabilities.

Bias analysis shows limited impact of race and gender manipulations.

Abstract

Objectives: Compare qualitative coding of instruction tuned large language models (IT-LLMs) against human coders in classifying the presence or absence of vulnerability in routinely collected unstructured text that describes police-public interactions. Evaluate potential bias in IT-LLM codings. Methods: Analyzing publicly available text narratives of police-public interactions recorded by Boston Police Department, we provide humans and IT-LLMs with qualitative labelling codebooks and compare labels generated by both, seeking to identify situations associated with (i) mental ill health; (ii) substance misuse; (iii) alcohol dependence; and (iv) homelessness. We explore multiple prompting strategies and model sizes, and the variability of labels generated by repeated prompts. Additionally, to explore model bias, we utilize counterfactual methods to assess the impact of two protected characteristics - race and gender - on IT-LLM classification. Results: Results demonstrate that IT-LLMs can effectively support human qualitative coding of police incident narratives. While there is some disagreement between LLM and human generated labels, IT-LLMs are highly effective at screening narratives where no vulnerabilities are present, potentially vastly reducing the requirement for human coding. Counterfactual analyses demonstrate that manipulations to both gender and race of individuals described in narratives have very limited effects on IT-LLM classifications beyond those expected by chance. Conclusions: IT-LLMs offer effective means to augment human qualitative coding in a way that requires much lower levels of resource to analyze large unstructured datasets. Moreover, they encourage specificity in qualitative coding, promote transparency, and provide the opportunity for more standardized, replicable approaches to analyzing large free-text police data sources.