Prompt-and-Check: Using Large Language Models to Evaluate Communication Protocol Compliance in Simulation-Based Training

TL;DR

This paper introduces Prompt-and-Check, a lightweight method using open-source large language models to evaluate communication protocol compliance in simulation training, demonstrating effective context-aware reasoning without task-specific training.

Contribution

The paper presents a novel prompt-based approach for assessing protocol compliance using open-source LLMs, enabling efficient evaluation solely from transcribed verbal exchanges.

Findings

Models achieved high agreement with expert annotations.

Prompting enabled effective context-aware reasoning.

Method ran efficiently on consumer-grade GPUs.

Abstract

Accurate evaluation of procedural communication compliance is essential in simulation-based training, particularly in safety-critical domains where adherence to compliance checklists reflects operational competence. This paper explores a lightweight, deployable approach using prompt-based inference with open-source large language models (LLMs) that can run efficiently on consumer-grade GPUs. We present Prompt-and-Check, a method that uses context-rich prompts to evaluate whether each checklist item in a protocol has been fulfilled, solely based on transcribed verbal exchanges. We perform a case study in the maritime domain with participants performing an identical simulation task, and experiment with models such as LLama 2 7B, LLaMA 3 8B and Mistral 7B, running locally on an RTX 4070 GPU. For each checklist item, a prompt incorporating relevant transcript excerpts is fed into the model, which outputs a compliance judgment. We assess model outputs against expert-annotated ground truth using classification accuracy and agreement scores. Our findings demonstrate that prompting enables effective context-aware reasoning without task-specific training. This study highlights the practical utility of LLMs in augmenting debriefing, performance feedback, and automated assessment in training environments.