Into the crossfire: evaluating the use of a language model to crowdsource gun violence reports

TL;DR

This study evaluates a fine-tuned BERT-based language model to assist Brazilian human rights analysts in monitoring social media for gun violence reports, improving efficiency and engagement.

Contribution

It introduces a novel application of a language model for real-time gun violence monitoring in social media, demonstrating practical benefits for human rights work.

Findings

Enhanced analyst efficiency and search capacity.

Increased interactions with social media users reporting gun violence.

Positive qualitative feedback from analysts.

Abstract

Gun violence is a pressing human rights issue that affects nearly every dimension of the social fabric, from healthcare and education to psychology and the economy. Reliable data on firearm events is paramount to developing more effective public policy and emergency responses. However, the lack of comprehensive databases and the risks of in-person surveys prevent human rights organizations from collecting needed data in most countries. Here, we partner with a Brazilian human rights organization to conduct a systematic evaluation of language models to assist with monitoring real-world firearm events from social media data. We propose a fine-tuned BERT-based model trained on Twitter (now X) texts to distinguish gun violence reports from ordinary Portuguese texts. We then incorporate our model into a web application and test it in a live intervention. We study and interview Brazilian analysts who continuously check social media texts to identify new gun violence events. Qualitative assessments show that our solution helped all analysts use their time more efficiently and expanded their search capacities. Quantitative assessments show that the use of our model was associated with analysts having further interactions with online users reporting gun violence. Our findings suggest that human-centered interventions using language models can help support the work of human rights organizations.