CliniDial: A Naturally Occurring Multimodal Dialogue Dataset for Team Reflection in Action During Clinical Operation

TL;DR

CliniDial is a multimodal dataset capturing natural team interactions during simulated medical procedures, designed to facilitate research on teamwork and improve AI models' understanding of complex clinical collaboration.

Contribution

The paper introduces CliniDial, a novel, richly annotated multimodal dataset of clinical team interactions during simulations, highlighting its challenges for current language models.

Findings

Existing models struggle with CliniDial's data complexity.

Rich, natural interactions pose challenges for AI understanding.

Multimodal data enhances the analysis of clinical teamwork.

Abstract

In clinical operations, teamwork can be the crucial factor that determines the final outcome. Prior studies have shown that sufficient collaboration is the key factor that determines the outcome of an operation. To understand how the team practices teamwork during the operation, we collected CliniDial from simulations of medical operations. CliniDial includes the audio data and its transcriptions, the simulated physiology signals of the patient manikins, and how the team operates from two camera angles. We annotate behavior codes following an existing framework to understand the teamwork process for CliniDial. We pinpoint three main characteristics of our dataset, including its label imbalances, rich and natural interactions, and multiple modalities, and conduct experiments to test existing LLMs' capabilities on handling data with these characteristics. Experimental results show that CliniDial poses significant challenges to the existing models, inviting future effort on developing methods that can deal with real-world clinical data. We open-source the codebase at https://github.com/MichiganNLP/CliniDial