Generative AI in simulation debriefings: an exploratory study using the Team-FIRST framework and qualitative feedback from simulation experts and learners

TL;DR

This study explores how generative AI can help in medical simulation debriefings by analyzing team communication and providing feedback, though it highlights the need for human oversight.

Contribution

The study introduces AI-generated teamwork reports using the Team-FIRST framework in medical simulations, capturing observations that might otherwise be missed.

Findings

Experts found AI reports useful for structured feedback and capturing overlooked observations.

Learners were optimistic about AI's potential but concerned about transparency and interpretation errors.

Limitations included inaccuracies in speaker attribution and lack of contextual/nonverbal data.

Abstract

Effective debriefings in simulation-based education require accurate observation of team interactions, yet facilitators face challenges due to cognitive load, observer bias, and the complexity of team dynamics. Generative artificial intelligence (AI) tools offer a potential means to support this process by analyzing verbal communication and providing structured feedback. This study explored how AI tools can contribute to teamwork observation and debriefing in immersive medical simulations.

We conducted a qualitative, exploratory study using thematic analysis of simulation participants’ and debriefers’ experiences with AI-generated teamwork reports. Forty-one participants (anesthesia nurses, residents, and attendings) participated in immersive scenarios at the University Hospital Zurich simulation center. Verbal interactions were transcribed with AI-assisted speech recognition and analyzed using two large language model–based systems (Isaac and ChatGPT-4o) guided by a prompt based on the Team-FIRST framework. Structured reports were generated for each scenario and reviewed by four simulation experts. Semi-structured interviews captured learners’ perspectives on being observed by AI tools.

A total of 26 AI-generated reports and 27 learner interviews were analyzed. Experts valued the detailed transcripts and illustrative quotes, which supported structured feedback and captured observations that might otherwise be missed. Limitations included inaccuracies in categorization, misattribution of speakers, overly generalized interpretations, and the absence of contextual or nonverbal information. Learners expressed openness and optimism about AI’s potential benefits: efficiency, objectivity, and enhanced perception, while also raising concerns about transparency, data protection, interpretation errors, and risks of overreliance. Both groups emphasized the necessity of human oversight.

Generative AI tools can complement simulation debriefings by structuring communication data and highlighting teamwork patterns, supporting reflective practice. Current limitations highlight the need for multimodal approaches, refined prompting strategies, and integration with expert facilitation to ensure AI functions as a support tool rather than a replacement in simulation-based education.

BASEC ID: Req-2024-01642.

The online version contains supplementary material available at 10.1186/s41077-026-00407-0.

Figures