LogiDebrief: A Signal-Temporal Logic based Automated Debriefing Approach with Large Language Models Integration

TL;DR

LogiDebrief is an AI framework that automates 9-1-1 call debriefing by combining Signal-Temporal Logic with Large Language Models, enabling comprehensive and efficient performance evaluations of emergency call responses.

Contribution

It introduces a novel integration of STL and LLMs for automated, systematic, and real-time evaluation of emergency call procedures, improving over traditional manual assessments.

Findings

Successfully deployed at Nashville emergency services, analyzing 1,701 calls.

Saved over 311 hours of manual review time.

Demonstrated high accuracy and effectiveness through empirical evaluation.

Abstract

Emergency response services are critical to public safety, with 9-1-1 call-takers playing a key role in ensuring timely and effective emergency operations. To ensure call-taking performance consistency, quality assurance is implemented to evaluate and refine call-takers' skillsets. However, traditional human-led evaluations struggle with high call volumes, leading to low coverage and delayed assessments. We introduce LogiDebrief, an AI-driven framework that automates traditional 9-1-1 call debriefing by integrating Signal-Temporal Logic (STL) with Large Language Models (LLMs) for fully-covered rigorous performance evaluation. LogiDebrief formalizes call-taking requirements as logical specifications, enabling systematic assessment of 9-1-1 calls against procedural guidelines. It employs a three-step verification process: (1) contextual understanding to identify responder types, incident classifications, and critical conditions; (2) STL-based runtime checking with LLM integration to ensure compliance; and (3) automated aggregation of results into quality assurance reports. Beyond its technical contributions, LogiDebrief has demonstrated real-world impact. Successfully deployed at Metro Nashville Department of Emergency Communications, it has assisted in debriefing 1,701 real-world calls, saving 311.85 hours of active engagement. Empirical evaluation with real-world data confirms its accuracy, while a case study and extensive user study highlight its effectiveness in enhancing call-taking performance.