Improving Emotional Support Delivery in Text-Based Community Safety Reporting Using Large Language Models

TL;DR

This paper analyzes two years of chat logs to understand emotional support in text-based incident reporting and introduces dispatcherLLM, a fine-tuned large language model, to enhance support quality and consistency.

Contribution

It provides empirical insights into emotional support delivery in dispatch chats and presents a novel fine-tuned LLM to improve support quality.

Findings

Significant variation in emotional support based on incident type and time

Support quality declined over time across organizations

dispatcherLLM outperforms baseline models in response quality

Abstract

Emotional support is a crucial aspect of communication between community members and police dispatchers during incident reporting. However, there is a lack of understanding about how emotional support is delivered through text-based systems, especially in various non-emergency contexts. In this study, we analyzed two years of chat logs comprising 57,114 messages across 8,239 incidents from 130 higher education institutions. Our empirical findings revealed significant variations in emotional support provided by dispatchers, influenced by the type of incident, service time, and a noticeable decline in support over time across multiple organizations. To improve the consistency and quality of emotional support, we developed and implemented a fine-tuned Large Language Model (LLM), named dispatcherLLM. We evaluated dispatcherLLM by comparing its generated responses to those of human dispatchers and other off-the-shelf models using real chat messages. Additionally, we conducted a human evaluation to assess the perceived effectiveness of the support provided by dispatcherLLM. This study not only contributes new empirical understandings of emotional support in text-based dispatch systems but also demonstrates the significant potential of generative AI in improving service delivery.